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(54) COMMUNICATION APPARATUS AND METHOD 



(57) The present invention makes It possible to 
process at high speed the foreground component image 
and the background component image of picked up im- 
ages on a networi( platfonn. 

A client computer 27 outputs infonmation specifying 
image data desired to separate to a separation server 
1 1 . The separation server 1 1 obtains the specified image 
data from a storage server 1 8 and outputs It to a motion 
detecting server 12 to perfonn motion detection 
processing. 

Thereafter, the image data, motion vector and po- 
sttionai information are output to an area specifying 



server 1 3. The area specifying server 1 3 generates area 
information of the image data and outputs the area in- 
fomiation to a mixture ratio calculating server 14 in ad- 
dition to the Image data, the motion vector and the po- 
sitional infonnatlon. The mixture ratio calculating server 
14 calculates a mixture ratio on the basis of the image 
data, the motion vector, the positional infonmation and 
the area infonnatlon, and a foreground/background im- 
age separation server 15 separates foreground and 
background of the Input Image on the basis of such in- 
fomiation. The present invention may be employed in a 
business model for image processing. 



FIG. 1 



!3 
I 



Q. 



MonoM 

OtTCCmlC 



7 • 


t 


. } } 




Stt¥EI 




UOA 












roRACt 
litem 



mTUtt[-«AT» 
CMCtnATM 



AOJUSTWC 

Stava 



tintn ( — 
wwvtoti 



2S>2 



Printsd by Jouvo, 76001 PARIS (FR) 



EP1400 927A1 



Description 
Technical Field 

[0001] The present invention relates to a communication system and a method thereof, and more particularly, to a 
communication system and a method for distributing image processing to a plurality of servers on a network to perfomn 
separated or isolated image processing desired by a user, wherein the communication system and the method make 
it possible to improve the speed of image processing on a network platfonm at reduced costs. 

Background Art 

[0002] in general, techniques for synthesizing images desired by a user on a network are widely known. 
[0003] Synthesized image is usually generated by overlapping and joining a number of pre-existing images such as 
images having already been stored in a server connected through a network or images having been made available 
from digital still cameras. In some cases, the synthesized image may be generated by a texture mapping. 
[0004] However, there are problems in that synthesized images generated by using the above conventional methods 
give rise to the occurrences of motion blur. For example, motion blur commonly occurs with a moving body that cannot 
be accurately adjusted into the synthesized images. Consequently, synthesizing processes are perfonmed with low 
accuracy and unnatural appearances are frequently generated in the synthesized images. 

[0005] Additionally, if one server performs synthesizing process in a multiple bundle task, a significant duration of 
time is required for the process. For example, even when the same background Images are to be overlapped, the 
overlapping process is repeated as many times as the number of images desired. This poses a problem to users in 
that when the processing services are charged by time, the cost can be a considerable burden. 

DISCLOSURE OF INVENTION 

[0006] In view of the foregoing, It is an object of the present invention to provide a communication system and a 
method for distributing image processing characterized by having multiple tasks to a plurality of servers on a networi< 
to pertonn separated or isolated Image processing desired by a user, wherein the communication system and the 
method make it possible to improve the speed of image processing on a network platfonm at reduced costs. 
[0007] A first communication apparatus of the present invention comprises a separating means for separating a 
predetennined image data that is obtained by an Image pickup element having a predetemnined number of pixels, each 
having a time Integration effect, and that is comprised of pixel values detenmined every pixel in accordance with an 
amount of light fomiing an image integrated overtime, into a foreground component Image having foreground object 
components constituting a foreground object and a background component image having background object compo- 
nents constituting a background object; a requested infomnation input means for inputting requested infonnatlon of a 
user; and an encoded data output means for outputting encoded data generated when the image is separated Into a 
foreground component image and a background component image by the separating means on the basis of the re- 
quested Infomnation input by the requested Infomnation input means. 

[0008] A charging means may be further comprised for perfomiing a charge processing in accordance with the re- 
quested Information. 

[0009] The charging means may generate charge infonnation including a user ID, a communication system ID and 
cost information corresponding to the requested information in accordance with the requested infonnation. 
[0010] The charging means may perfomn a charge processing with respect to a financial account of the user on the 
basis of the charge information. 

[0011] The charging means may perform the charge processing by deducting the number of points corresponding 
to the cost information from the number of points of each user used in the charge processing. 
[0012] The encoded data output means may output the encoded data in a way obtainable only to a user having 
finished the charge processing after the charge processing has been finished by the charging means. 
[0013] The requested infonnation input means may input a predetennined image data In addition to the requested 
infonnation of a user. The encoded data output means may output the encoded data as a significant infonnation gen- 
erated when the image is separated into the foreground component Image and the background component image by 
the separating means on the basis of the requested infonnation input by the requested information Input means and 
the predetermined image data, in a way obtainable only to the user having finished the charge processing after the 
charge processing has been finished by the charging means. 

[0014] An area Infonnation generating means for generating area information for discerning any of a foreground area 
having foreground object components constituting the foreground object of the predetennined image data, a back- 
ground area having background object components constituting the background object of the predetermined Image 
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data and a mixed area in which the foreground area and the baclcground area are mixed may be further comprised. 
The significant information may include the area information and the encoded data output means may output the 
encoded data as the area Information generated when the image is separated into the foreground component image 
and the baclcground component Image by the separating means on the basis of the requested Infomiation input by the 
requested infonmatlon Input means and the predetermined image data, in a way obtainabie only to the user after the 
charge processing has been finished by the charging means. 

[001 5] A mixture ratio generating means for generating a mixture ratio Indicating a ratio In which the foreground area 
and the bacl<ground area are mixed in the mixed area of the predetermined image data may be further comprised. The 
significant infomnation may include the mixture ratio and the encoded data output means may output the encoded data 
as the mixture ratio generated when the image is separated into the foreground component image and the baclcground 
component Image by the separating means on the basis of the requested information input by the requested infonmation 
Input means and the predetenmined image data, in a way obtainable only to the user after the charge processing has 
been finished by the charging means. 

[001 6] The separating means may separate a predetermined image data that Is obtained by an image piclcup element 
having a predetemnined number of pixels, each having a time integration effect, and that is comprised of pixel values 
detenmined every pixel in accordance with an amount of light fomning a predetemilned image integrated overtime, into 
a foreground component image having foreground object components constituting the foreground object and a baclc- 
ground component image having baclcground object components constituting the background object on the basis of 
the area Infomnation and the mixture ratio. The significant Infomnation may Include the foreground component image 
and the baclcground component image and the encoded data output means may output the requested infonmation Input 
by the requested infomnation input means and the encoded data as the foreground component image and the bacl<- 
ground component image generated when the image is separated into the foreground component image and the back- 
ground component image by the separating means, in a way obtainable only to the user after the charge processing 
has been finished by the charging means. 

[0017] The requested infomnation input means may input an Image data ID for disceming a predetemiined image 
data In addition to the requested Information of a user, and the encoded data output means may output the encoded 
data as the significant information generated when the predetermined image Is separated into the foreground compo- 
nent image and the background component image by the separating means on the basis of the requested Information 
input by the requested infomnation input means and the predetenmined image data corresponding to the image data 
ID, in a way obtainable only to the user after the charge processing has been finished by the charging means. 
[0018] An area infonmation generating means for generating area Information for disceming any of a foreground area 
having foreground object components constituting the foreground object of the predetermined Image data correspond- 
ing to the image data ID, a background area having background object components constituting the background object 
of the predetermined image data conresponding to the image data ID and a mixed area in which the foreground area 
and the background area are mixed may be further comprised. The significant infomnation may Include the area infor- 
mation and the encoded data output means may output the encoded data as the area infomnation generated when the 
predetemnined image is separated into the foreground component Image and the background component image by 
the separating means on the basts of the requested infomnation input by the requested information input means and 
the predetemnined image data, in a way obtainable only to the user after the charge processing has been finished by 
the charging means. 

[001 9] A mixture ratio generating means for generating a mixture ratio of the mixed area In the predetemnined Image 
data corresponding to the Image data ID may be further comprised. The significant infonnation may include the mixture 
ratio and the encoded data output means may output the encoded data as the mixture ratio generated when the pre- 
detenmined image is separated into the foreground component image and the background component image by the 
separating means on the basis of the requested infonnation input by the requested information input means and the 
predetemnined image data corresponding to the image data ID, in a way obtainable only to the user after the charge 
processing has been finished by said charging means. 

[0020] The separating means may separate a predetemiined image data that is obtained by an image pickup element 
having a predetemnined number of pixels, each having a time Integration effect, and that is comprised of pixel values 
detemnined every pixel in accordance with an amount of light forming an Image Integrated overtime, into a foreground 
component image having foreground object components constituting the foreground object and a background compo- 
nent image having background object components constituting the background object on the basis of the area infor- 
mation and the mixture ratio. The signiftoant information may Include the foreground component image and the back- 
ground component image and the encoded data output means may output the requested infonmation Input by the 
requested information input means and the encoded data as the foreground component image and the background 
component Image generated when a predetemnined Image is separated Into the foreground component image and the 
background component image by the separating means, in a way obtainable only to a user after the charge processing 
has been finished by the charging means. 
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[0021] A first communication method of the present invention comprises: a separating step of separating a prede- 
termined image data that is obtained by an image pickup element having a predetermined number of pixels, each 
having a time integration effect, and that is comprised of pixel values detemnined every pixel in accordance with an 
amount of light forming an image integrated over time, into a foreground component image having foreground object 

5 components constituting a foreground object and a background component image having background object compo- 
nents constituting a background object; a requested information input step of inputting requested infomnation of a user; 
and an encoded data output step of outputting encoded data generated when an image is separated into the foreground 
component image and the background component image in the separating step on the basis of the requested infor- 
mation input in the requested infonrtation input step. 

10 [0022] A program of a first recording medium of the present invention comprises: a separation control step of con- 
trolling separation of an image data that is obtained by an image pickup element having a predetemiined number of 
pixels, each having a time Integration effect, and that is comprised of pixel values detemnined every pixel in accordance 
with an amount of light fomning an image integrated overtime, into a foreground component image having foreground 
object components constituting a foreground object and a background component image having background object 

'5 components constituting a background object; a requested infomnation Input control step of controlling input of request- 
ed infomnation of a user; and an encoded data output control step of controlling output of encoded data generated 
when an image is separated Into the foreground component Image and the background component image in the sep- 
aration control step on the basis of the requested information input in the requested information Input control step. 
[0023] A first program of the present Invention makes a computer perfomn: a separation control step of controiiing 

20 separation of a predetennined image data that is obtained by an Image pickup element having a predetermined number 
of pixels, each having a time integration effect, and that is comprised of pixel values detemnined every pixel in accord- 
ance with an amount of light fomning an image integrated over time, into a foreground component image having fore- 
ground object components constituting a foreground object and a background component image having background 
object components constituting a background object; a requested information input control step of controlling input of 

25 requested information of a user; and an encoded data output control step of controiiing output of encoded data gen- 
erated when an image is separated Into the foreground component image and the background component Image in 
the separation control step on the basis of the requested infomnation input In said requested information Input control 
step. 

[0024] In the first communication system, the method thereof and the first program, an image data that is obtained 
30 by an image pickup element having a predetermined number of pixels, each having a time integration effect, and that 
is comprised of pixel values detenmined every pixel in accordance with an amount of light forming an image integrated 
over time is separated into a foreground component image having foreground object components constituting a fore- 
ground object and a background component Image having background object components constituting a background 
object; requested information of a user Is Input; and encoded data generated when an Image is separated into the 
35 foreground component image and the background component image on the basis of the requested information Input 
is output. 

Brief Description of the Drawings 
40 [0025] 

FIG. 1 Is a configuration view of an embodiment of an image processing system to which the present Invention is 
applied; 

FIG. 2 is a configuration view of a separation server in FIG. 1 ; 
45 FIG. 3 is a configuration view of a camera terminal unit in FIG. 1 ; 

FIG. 4 is a functional view of the separation server in FIG. 1 ; 

FIG. 5 is another functional view of the separation server in FIG. 1 ; 

FIG. 6 is a functional view of a motion detecting server In FIG. 1 ; 

FIG. 7 is another functional view of the motion detecting server in FIG. 1 ; 
50 FIG. 8 is a functional view of an area specifying server in FIG. 1 ; 

FIG. 9 is another functional view of the area specifying server in FIG. 1 ; 

FIG. 10 is a functional view of a mixture ratio calculating server in FIG. 1 ; 

FIG. 11 is another functional view of the mixture ratio calculating server In FIG. 1 ; 

FIG. 12 is a functional view of a foreground/background image separating server In FIG. 1 ; 
S5 FIG. 13 Is another functional view of the foreground/background Image separating server in FIG. 1 ; 

FIG. 14 is a functional view of a motion blur adjusting server in FIG. 1 ; 

FIG. 15 is another functional view of the motion blur adjusting server in FIG. 1 ; 

FIG. 16 Is a functional view of an encoding server In FIG. 1 ; 
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FIG. 1 7 is another functional view of the encoding server In FiQ. 1 ; 

FIG. 18 is a functional view of a storage server in FIG. 1 ; 

FIG. 19 Is another functional view of the storage server In FIG. 1 ; 

FIG. 20 is a functional view of a synthesizing server In FIG. 1; 
5 FIG. 21 is another functional view of the synthesizing server in FIG. 1 ; 

FIG. 22 is a functional view of a correcting server in FIG. 1 ; 

FIG. 23 is another functional view of the correcting server in FIG. 1 ; 

FIG. 24 is a functional view of a purchasing server in FIG. 1 ; 

FIG. 25 is a functional view of a selling server in FIG. 1 ; 
10 FIG. 26 is a functional view of a retrieving server in FIG. 1 ; 

FIG. 27 is a blocl^ diagram showing the separation server; 

FtG. 28 is a diagram illustrating image pickup by a sensor; 

FIG. 29 is a diagram illustrating an arrangement of pixels; 

FIG. 30 is a diagram illustrating operation of a detecting element; 
IS FIG. 31 A is a diagram illustrating an image obtained by picking up an image of an object conBsponding to a moving 

foreground and an object con^esponding to a stationary background; 

FIG. 31 B is another diagram illustrating an image obtained by picking up an image of an object corresponding to 
a moving foreground and an object corresponding to a stationary background; 

FIG. 32 is a diagram illustrating background area, foreground area, mixed area, covered background area and 
20 uncovered background area; 

FIG. 33 is a modeling diagram in whteh pixel values of pixels adjacent to each other In a row are expanded in time 
direction in picking up an image of an object corresponding to a stationary foreground and an object corresponding 
to a stationary background; 

FIG. 34 IS a modeling diagram in which pixel values are expanded in the time direction and shutter time which is 
25 divided into a time interval; 

FIG. 35 is a modeling diagram in which pixel values are expanded In the time direction and shutter time which is 

divided into a time interval; 

FIG. 36 is a modeling diagram in which pixel values are expanded In the time direction and shutter time which is 
divided into a time Interval; 

30 FIG. 37 Is a diagram illustrating an example In whteh pixels of foreground area, background area and mixed area 

are extracted; 

FIG. 38 Is a diagram illustrating relationship of pixels and a model in which pixel values are expanded In the time 

direction; 

FIG. 39 is a modeling diagram 
35 divided into a time interval; 

FiG. 40 is a modeling diagram 

divided into a time Interval; 

FIG. 41 is a modeling diagram 

divided into a time Interval; 
40 FIG. 42 is a modeling diagram 

divided into a time Interval; 

FIG. 43 is a modeling diagram 

divided into a time interval; 

FIG. 44 is a flowchart illustrating a process of adjusting the amount of motion blur. 
45 FIG. 45 is a block diagram illustrating a configuration example of an area specifying unit 103; 

FIG. 46 Is a diagram Illustrating an image when an object con^esponding to foreground is being moved; 

FIG. 47 is a modeling diagram in which pixel values are expanded in the time direction and shutter time which is 

divided Into a time interval; 

FIG. 48 is a modeling diagram in which pixel values are expanded in the time direction and shutter time which is 
50 divided into a time interval; 

FIG. 49 is a modeling diagram in which pixel values are expanded in the time direction and shutter time which Is 
divided Into a time interval; 

FIG. 50 Is a diagram Illustrating conditions of an area determination; 

FIG. 51 A is a diagram illustrating an example of an area specification result in the area specifying unit 103; 
55 FIG. 51 B is a diagram Illustrating an example of an area specification result in the area specifying unit 1 03; 

FIG. 61 C is a diagram illustrating an example of an area specification result In the area specifying unit 103; 
FIG. 51 D is a diagram illustrating an example of an area specification result in the area specifying unit 103; 
FIG. 52 is a diagram illustrating an example of an area speclfk:atlon result in the area specifying unit 103; 



in which pixel values are expanded in the time direction and shutter time which Is 
in which pixel values are expanded in the time direction and shutter time whicfi is 
In which pixel values are expanded in the time direction and shutter time which is 
in which pixei values are expanded In the time direction and shutter time which is 
in which pixel values are expanded in the time direction and shutter time which is 
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FIG. 53 is a flowchart illustrating an area specification processing; 

FIG. 54 is a block diagram of another example of a configuration of the area spedfying unit 103; 

FIG. 55 is a modeling diagram In which pixel values are expanded in the time direction and shutter time which Is 

divided into a time interval; 

FIG. 56 is a diagram illustrating an example of a background image; 

FIG. 57 is a block diagram illustrating a configuration of a binary object Image extracting portion 302; 

FIG. 58A is a diagram illustrating cak^utation of correlation value; 

FIG. 58B Is a diagram Illustrating catoulation of conrelatlon value; 

FIG. 59A is a diagram Illustrating calculation of correlation value; 

FIG. 59B Is a diagram illustrating calculation of correlation value; 

FIG. 60 is a diagram illustrating an example of a binary object image; 

FIG. 61 is a block diagram illustrating a configuration of a time change detector 303; 

FIG. 62 is a diagram illustrating a determination in an area determining portion 342; 

FIG. 63 is a diagram illustrating an example of a determination in the time change detector 303; 

FIG. 64 is a flowchart illustrating an area specification processing in the area detemnining unit 103; 

FIG. 65 is a flowchart Illustrating details of the area specification processing; 

FIG. 66 is a block diagram of another configuration of the area specifying unit 103; 

FIG. 67 is a block diagram illustrating a configuration of a robust processing portion 361 ; 

FIG. 68 Is a diagram illustrating motion compensation in a motion compensator 381; 

FIG. 69 Is a diagram illustrating motion compensation in the motion compensator 381 ; 

FIG. 70 Is a flowchart Illustrating the area specification processing; 

FIG. 71 is a flowchart illustrating details of a robust processing; 

FIG. 72 is a block diagram illustrating an example of a configuration of a mixture ratio calculating unit 104; 
FIG. 73 is a diagram illustrating an example of an Ideal mixture ratio a; 

FIG. 74 is a modeling diagram In which pixel values are expanded in the time direction and shutter time which Is 
divided into a time interval; 

FIG. 75 is a modeling diagram In which pixel values are expanded in the time direction and shutter time which is 
divided Into a time interval; 

FIG. 76 is a diagram illustrating an approximation using correlation of foreground components; 
FIG. 77 Is a diagram Illustrating the relationship of C, N and P; 

FIG. 78 is a block diagram illustrating a configuration of an estimated mixture ratio processor 401 ; 

FIG. 79 Is a diagram illustrating an example of an estimated mixture ratio; 

FIG. 80 is a block diagram illustrating another configuration of the mixture ratio calculator 104; 

FIG. 81 is a flowchart Illustrating a mixture ratio calculation processing; 

FIG. 82 is a flowchart Illustrating an estimated mixture ratio calculation processing; 

FIG. 83 is a diagram illustrating a linear line approximation of the mixture ratio a; 

FIG. 84 Is a diagram Illustrating a plane approximation of the mixture ratio a; 

FIG. 85 Is a diagram illustrating conrespondence of pixels In a plurality frames when the mixture ratio a is calculated; 
FIG. 86 is a block diagram Illustrating another configuration of the mixture ratio estimating processor 401 ; 
FIG. 87 is a diagram Illustrating an example of an estimated mixture ratio; 

FIG. 88 is a flowchart illustrating the mixture ratio estimation processing by using a model con-espondlng to a 
covered background area; 

FIG. 89 is a block diagram illustrating an example of a configuration of a foreground/background separator 105; 
FIG. 90A Is a diagram showing an input image, a foreground component Image and a background component 
image; 

FIG. 90B is a diagram showing an input Image, a foreground component image and a background component 
image; 

FIG. 91 is a modeling diagram in which pixel values are expanded in the time direction and shutter time which is 
divided Into a time interval; 

FIG. 92 is a modeling diagram In which pixel values are expanded in the time direction and shutter time which is 
divided into a time interval; 

FIG. 93 is a modeling diagram in which pixel values are expanded in the time direction and shutter time which is 
divided Into a time interval; 

FIG. 94 is a block diagram illustrating an example of a configuration of a separating portion 601 ; 

FIG. 95A Is a diagram showing an example of a foreground component image and a background component Image 

separated; 

FIG. 958 is a diagram showing an example of a foreground component image and a background component image 
separated; 



6 



EP 1 400 927 A1 



FIG. 96 Is a flowchart Illustrating a foreground/background separation processing; 

FIG. 97 is a block diagram showing an example of a configuration of a motion blur adjusting unit 106; 

FIG. 98 is a diagram illustrating a unit of processing; 

FIG. 99 is a modeling diagram In whteh pixel values of a foreground component Image are expanded In the time 
5 direction and shutter time which is divided into a time interval; 

FIG. 100 is a modeling diagram in which pixel values of a foreground component image are expanded in the time 
direction and shutter time which is divided into a time interval; 

FIG. 101 is a modeling diagram in which pixel values of a foreground component Image are expanded In the time 
direction and shutter time which is divided into a time interval; 
10 FIG. 1 02 is a modeling diagram in which pixel values of a foreground component image are expanded In the time 

direction and shutter time which is divided into a time interval; 

FIG. 1 03 is a diagram showing another configuration of the motion blur adjusting unit 1 06; 
FIG. 104 is a flowchart illustrating a process of adjusting the amount of motion blur contained in a foreground 
component image in the motion blur adjusting unit (106); 
15 FIG. 1 05 is a block diagram showing another example of the configuration of Uie motion blur adjusting unit 1 06; 

FIG. 106 is a diagram showing an example of a model in which the relationship of pixel values and foreground 
components is specified; 

FIG. 107 is a diagram illustrating a calculation of a foreground component; 
FIG. 108 is a diagram illustrating a calculation of a foreground component; 
20 FIG. 1 09 is a flowchart illustrating the motion blur elimination processing of a foreground; 

FIG. 110 is a block diagram showing another configuration of a function of the separation server; 
FIG. Ill is a diagram showing a configuration of a synthesizer 1001 ; 

FIG. 11 2 is a block diagram showing another configuration of a function of the separation server; 

FIG. 113 is a block diagram showing a configuration of a mixture ratio calculator 11 01 ; 
25 FIG. 114 is a block diagram showing a configuration of a foreground/background separator 1102; 

FIG. 1 1 5 is a block diagram showing another configuration of a function of the separation server; 

FIG. 116 is a diagram showing a configuration of a synthesizer 1201 ; 

FIG. 117 is a flowchart Illustrating a separating service; 

FIG. 118 is a flowchart illustrating a charge processing; 
30 FIG. 119 Is a diagram illustrating a charge processing; 

FIG. 120 is a flowchart illustrating another example of the charge processing; 

FIG. 121 Is a flowchart Illustrating a motion detecting service; 

FIG. 122 is a flowchart illustrating an area specifying service; 

FIG. 123 is a flowchart Illustrating a mixture ratio calculating servbe; 
35 FIG. 1 24 is a flowchart illustrating a foreground/background separating service; 

FIG. 125 is a flowchart illustrating a motion blur adjusting service; 

FIG, 126 is a diagram illustrating an encoding server; 

FIG. 127 is a flowchart illustrating an encoding service; 

FIG. 128 is a diagram illustrating a compressing ability by a means of the encoding processing; 
40 FIG. 129 is a diagram illustrating another example of the encoding server; 

FIG. 130 is a flowchart illustrating a synthesizing service; 

FIG. 131 is a diagram Illustrating an encrypting motion blur adding unit; 

FIG. 132 is a diagram illustrating an encrypting motion blur eliminating unit; 

FIG. 133 is a diagram illustrating a process of adding an encrypting motion blur; 
45 FIG. 134 is a diagram illustrating a process of adding an encrypting motion blur; 

FIG. 135 is a diagram Illustrating a process of adding an encrypting motion blur; 

FIG. 136 is a flowchart illustrating an encryption processing; 

FIG. 137 is a diagram illustrating a process of adding an encrypting motion blur; 

FIG. 138 is a diagram illustrating a process of adding an encrypting motion blur; 
50 FIG. 139 Is a diagram Illustrating a process of adding an encrypting motion blur; 

FIG. 140 is a diagram illustrating a process of adding an encrypting motion blur; 

FIG. 141 is a diagram Illustrating a process of adding an encrypting motion blur; 

FIG. 142 is a diagram illustrating a process of adding an encrypting motion blur; 

FIG. 143 is a diagram illustrating a process of adding an encrypting motion blur; 
55 FIG. 144 is a diagram illustrating a process of adding an encrypting motion blur; 

FIG. 145 is a diagram illustrating a process of adding an encrypting motion blur; 

FIG. 146 is a diagram Illustrating a process of adding an encrypting motion blur; 

FIG. 147 is a diagram Illustrating a process of adding an encrypting motion blur; 
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FIG. 148 is a diagram illustrating a process of adding an encrypting motion biur; 

FIG. 149 is a diagram illustrating a process of adding an encrypting motion blur; 

FIG. 150 is a diagram illustrating a process of adding an encrypting motion blur; 

FIG. 151 is a flowchart illustrating an encrypting service; 
5 FIG. 152 Is a diagram illustrating a correcting server; 

FIG. 153A Is a diagram Illustrating an adjusting process; 

FIG. 153B is a diagram illustrating an adjusting process; 

FIG. 153C is a diagram illustrating an adjusting process; 

FIG. 153D is a diagram illustrating an adjusting process; 
10 FIG. 154 is a flowchart illustrating an adjusting process; 

FIG. 155 is a flowchart illustrating a purchasing service; 

FIG. 156 is a flowchart illustrating a selling service; 

FIG. 157 is a flowchart illustrating a charging service; 

FIG. 158 is a diagram illustrating the retrieving server; 
15 FIG. 159 is a flowchart illustrating a retrieving service; and 

FIG. 160 is a diagram illustrating a retrieving service. 

Best Mode for Carrying Out the Invention 
20 [0026] 

FIG. 1 is a drawing showing an embodiment of an Image processing system according to the present Invention. 

The image processing system of the present invention is constructed such that a separation server 1 1 , a motion 
detecting server 12, an area specifying server 13, a mixture ratio calculating server 14, a foreground/background 

25 image separating server 1 5, a motion blur adjusting server 1 6, an encoding server 1 7, storage servers 18-1,1 8-2, 

a synthesizing server 19, a con-ecting server 20, a purchasing server 21 , a selling server 22, a retrieving server 
23, an account charge server 24, a financial transaction server (for customer) 25, a financial transaction server 
(for provider) 26, client computers 27 and camera ternilnai units 28-1 - 28-n are connected over a network 1 such 
as the Internet to mutually exchange data. The separation server 11 , the motion detecting server 12, the area 

30 specifying server 13. the mixture ratio calculating server 14, the foreground/background Image separating server 
15, the motion blur adjusting server 16, the encoding server 17, the synthesizing server 19, the correcting server 
20, the purchasing server 21 , the selling server 22, the retrieving server 23, the account charge server 24 and the 
financial transaction servers (for customer and for provider) 25, 26 are managed or administrated by a provider 
who provides separating service, motion detecting service, area specifying service, mixture ratio calculating serv- 

35 ice, foreground/background separating service, motion blur adjusting service, encoding service, synthesizing serv- 

ice, correcting service, purchasing service, selling service, retrieval service, charging service and financial trans- 
action servtees (for client and provider), respectively Further^ In the following description, the storage servers 18-1, 
1 8-2 or the camera tenminal units 28-1 to 28-n are simply named the storage server 1 8 or the camera terminal unit 
28 when there is no need to distinguish them. Further, it is true of other servers and units. 

40 FIG. 2 is a configuration view of the separation server 11 of the present invention. 

A CPU (Central Processing Unit) 41 performs various kinds of processing in accordance with programs stored 
In a ROM (Read Only Memory) 42 or a storage unit 48. Data or programs executed by CPU 41 are stored In a 
RAM (Random Access Memory) 43 as needed. These CPU 41 , ROM 42 and RAM 43 are connected to one another 
via a bus 44. 

45 An input/output interface 45 is connected to CPU 41 via bus 44. An input unit 46 including a keyboard, a 

mouse, microphone, etc. and an output unit 47 including a display, speakers, etc. are connected to the input/output 
interface 45, The CPU 41 performs various kinds of processing according to instnjctions input from the input unit 
46. The CPU 41 outputs images, voices, etc. obtained as a result of processing to the output unit 47. 

The storage unit 48 connected to the input/output interface 45 comprises, for example, a hard disk and stores 
50 programs executed by CPU 41 or various kinds of data. A communication unit 49 communicates with external 
devices via the Internet or other networks. 

Further, programs may be obtained through the communication unit 49 and be stored in a storage unit 48. 
When a magnetic disk 61 , an optical disk 62, an optical magnetic disk 63, a semiconductor memory, etc. are 
mounted, a drive 50 connected to the input/output interface 45 drives them to obtain programs or data stored 
55 therein. The obtained programs or data are transfen-ed and stored in the storage unit 48 as needed. 

Further, since the basic configuration of the motion detecting server 12, the area specifying server 13, the 
mixture ratio calculating server 14, the foreground/background image separating server 15, the motion blur ad- 
justing server 1 6, the encoding server 1 7, the storage servers 18-1,1 8-2, the synthesizing server 1 9, the connecting 
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server 20, the purchasing server 21 , the selling server 22, the retrieving server 23, the account charge server 24, 
the financial transaction server (for customeO 25, the financial transaction server (for provider) 25 and the client 
computers 27 are the same as that of the separation sender 11 , the description thereof will be omitted. 
FIG. 3 is a drawing showing a configuration of the camera tenninal unit 28 of the present Invention. The camera 

5 temilnal unit 28 is constructed such that a sensor 76a and a GPS (Global Positioning System) 76b are provided 

In an input unit 76, an LCD (Liquid Crystal Display) 77a Is provided In an output unit 77 and the other structures 
are the same as that of the separation server 11. That is, a CPU 71 , a ROM 72, a RAM 73, a bus 74, an Input/ 
output interface 75, an input unit 76, an output unit 77, a storage unit 78, a communication unit 79. a drive 80. a 
magnetic disk 91 . an optical 6\sk 92, an optical magnetic dislc 93 and a semiconductor memory 94 of the camera 

10 teoDinal unit 28 conrespond to the CPU 41, the ROM 42, the RAM 43. the bus 44, the Input/output Interface 45. 

the Input unit 46. the output unit 47. the storage unit 48, the communication unit 49, the drive 50, the magnetic 
disk 61 , the optical disk 62, the optical magnetic disk 63 and the semiconductor memory 64 of the separation 
server 11, respectively. 

The sensor 76a Is an image pickup element and outputs the picked up image to the input unit 76. The GPS 
15 76b detects positional Infonnation on earth (latitude and longitude) based on signals transmitted from a stationary 

satellite (not shown) and outputs the detected positional infonnation to the input unit 76. The LCD 77a displays 
the image output from the output unit 77. 

[0027] Next, referring to FIGs. 4 and 5, the separation server 11 will be described. 

20 [0028] As shown In FIG. 4, the separation server 1 1 separates an image input from, for example, the client computers 
27 via the network 1 , Into a foreground component Image and a background component image by means of the method 
to be mentioned later, generates ID for the input Image, the foreground component Image and the background com-, 
ponent image and then, outputs them to the client computers 27, stores them In Its own storage unit, outputs them to 
the storage server 18 to be stored, or outputs them to other servers on the network to be stored via the network 1 . 

25 Here, the foreground component Image means an Image having a moving component of the Input image and the 
background component Image means an image of stationary part not having the moving component. At this time, a 
charge processor 11a makes the account charge server 24 perfomn charge processing for the cost of the separation 
processing via the network 1 . Further, as shown In FIG. 5, if an Image ID specifying an image Is input in place of an 
image, the separation server 11 accesses the retrieving server 23 to be described later or the storage server 18 on 

30 the networi< 1 or retrieves its own storage unit (for example, the storage unit 78 of FIG. 3) to read the image data 
corresponding to the input Image ID and separates the image data into a foreground component image and a back- 
ground component image. Thereafter, the separation sender 1 1 generates ID for each image, which is stored In its own 
storage unit, or outputs them to other servers on the network 1 so as to perfonn their respective processing. 
[0029] Further, in the following description, the image ID is explained as an example of Image specifying infonnation. 

35 However, any other Infonnation capable of specifying an Image, for example, positional infonnation of Image to be 
described later may be used as the image specifying Information. 

[0030] Next, referring to the FIGs. 6 and 7, the motion detecting server 12 will be explained. 

[0031] As shown in FIG. 6, an object extracting unit 12a of the motion detecting server 12 extracts, for example, an 

Image object of Image Input from the client computer 27, etc. and outputs them to a motion detector 1 2b. The motion 

40 detector 1 2b detects the motion vector and the positional infonnation of the input image object and then, outputs them 
the client computers 27, stores them In its own storage unit or outputs them to other servers on the network 1 so as 
to perform their respective processing. At this time, a charge processor 12c makes the account charge server 24 
perfonn the charge processing of cost for detecting the motion vector and the positional Infonnation of each image 
object via the network 1. An image that is a target of Image pickup and corresponds to an object In the real worid Is 

45 refen-ed to as an object In this specification. 

[0032] Also, as shown In FIG. 7, If an image ID specifying an image Is Input Instead of the image, the motion detecting 
server 1 2 accesses the retrieving server 23 to be described later or the storage server 1 8 on the networi^ 1 or retrieves 
its own storage unit (for example, the storage unit 78 of FIG. 3) to read out the image data con-esponding to the Input 
image ID and then, performs the same processing as mentioned above. 

so [0033] Next, referring to FIGs. 8 and 9, the area specifying server 13 will be explained. 

[0034] As shown In FIG. 8, the area specifying server 13 specifies the foreground area, the background area, the 
mixed area with respect to each pixel of the input image on the basis of an image Input from the client computer 27 
via the network 1 , and object specifying Information specifying object of the input Image, generates Infonnation (here- 
inafter, refen-ed to as area infonnation) specifying which pixel is included of the foreground area, the background area 

55 and the mixed area and then, outputs It to the client computer 27, stores It In Its own storage unit or outputs It to other 
servers on the network 1 so as to perfonn their respective processing. At this time, a charge processor 13a makes the 
account charge server 24 perform the charge processing of cost for the area specification processing via the networi< 
1 . Further, as shown in FIG. 9, If an Image ID specifying an Image is Input Instead of the image, the area specifying 
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[0044] As shown in FIG. 1 8, the storage sen/er 1 8 is connected via the networic 1 , stores the image transmitted from 
various icinds of servers, and outputs an image positional infomiation con^esponding to the stored image along with an 
image ID. For example, the client computer 27 can access via the network 1 and call out the desired image using the 
image positional information. That is, as shown in FIG. 1 9, for example, the client computer 27 can access the storage 

5 server 1 8 on the network 1 on the basis of the image positional infomiation and read out the desired image by specifying 
the image ID conresponding to the desired image. Further, the image positional information and the Image ID are 
explained separately in this speclfteation but the image positional information may be a part of the image ID. In this 
case, the image positional infonnation and the server on the network 1 in which the image positional infonnatlon Is 
stored (memorized or processed) can be recognized from the image ID. Further, the storage server 18 may store the 

10 motion vector, the positional infonnation, the mixture ratio and the amount of motion blur as well as the image data. 
[0045] Next, referring to FIGs. 20 and 21 , the synthesizing server 19 will be explained. 

[0046] As shown in FIG. 20, the synthesizing sen/er 1 9 synthesizes images A and B by using two images A, B input 
from, for example, the client computer 27, etc., the motion vector, the positional information, the mixture ratio and the 
amount of motion blur to generate a synthesized image A+B and then, outputs them to the client computer 27, stores 

IS it in its own storage unit or outputs it to other servers on the networi< so as to perfonn their respective processing via 
the networic 1 . In this case, the images A, B are synthesized by treating one of the images as the foreground component 
image and the other as the background component image. At this time, a charge processor 19a makes the account 
charge server 24 perform the charge processing of cost for the synthesizing process via the network 1 . 
Further, as shown in FIG. 21 , if an image A ID and an image B ID specifying the image A and B are input instead of 

20 the images A and B. the synthesizing server 1 9 accesses the retrieving server 23 to be described later, the storage 
server 1 8 on the networic or retrieves Its own storage unit (for example, the storage unit 78 of FIG. 3) to call out the 
images corresponding to the input image A ID and image B ID and then, performs the same processing as mentioned 
above. 

[0047] Next, referring to FIGs. 22 and 23, the correcting server 20 will be explained. 

25 [0048] As shown in FIG. 22, the correcting server 20 con-ects the image Input from, for example, the client computer 
27, etc. on the basis of the motion vector, the positional infonnation, the mixture ratio and the amount of motion blur, 
generates and outputs the corected image to the client computer 27, to store its own storage unit or to other servers 
on the networic 1 so as to perfomn their respective processing. At this time, the charge processor 20a makes the account 
charge server 24 perform the charge processing of cost for the correction processing via the networic 1 . Further, as 

30 shown in FIG. 23, If an Image ID specifying the image is input instead of the image, the correcting server 20 accesses 
the retrieving server 23 to be described later or the storage server 18 or retrieves its own storage unit (for example, 
the storage unit 78 of FIG. 3) to call out the Image corresponding to the input image ID and then, performs the same 
processing as mentioned above. 

[0049] Next, referring to FIG. 24, the purchasing server 21 will be explained. 

35 [0050] As shown In FIG. 24, If an image ID specifying an image desired to be purchased is input from, for example, 
the client computer 27, etc. by a user who wants to purchase the image, the purchasing server 21 accesses the sep- 
aration server 11 , the storage server 18, the synthesizing server 19 or the correcting server 20 on the networic 1 to call 
out the corresponding image and outputs that image to, for example, the client computer 27 via the networic. At this 
time, the charge processor 21a makes the account charge server 24 perfonn the charge processing of cost for the 

40 purchasing processing via the network 1 . 

[0051] Next, referring to FIG. 25, the selling server 22 will be explained. 

[0052] As shown in FIG. 25, If an image which may be a separated image, a synthesized image or a con-ected image 
generated, for example, by means of the separation server 11 , the synthesizing server 19 or the correcting server 20 
is input via the networic 1 by a user who wants to sell the image, the selling server 22 stores the image in the separation 

45 server 11 , the storage server 18, the synthesizing server 19 or the correcting server 20 on the networic 1, and the 
charge processor 22a makes the account charge server 24 perform the charge processing of cost for the sold image 
via the networic 1 (in this case, the provider of processing of the selling service make the user desiring the sale perfonn 
the payment processing of the processing fee corresponding to the sold image). 
[0053] Next, referring to FIG. 26, the retrieving server 26 will be explained, 

50 [0054] The retrieving sen/er 26 retrieves an image which Is being picked up or has been picked up by the camera 
terminal units 28-1 to 28-n on the networic 1 on the basis of information specifying the features of an image desired by 
a user from the client computer 27 and the physical positional Infonnation of the camera temninal units 28-1 to 28-n to 
output it to, for example, the client computer 27 as the requested image. At this time, the charge processor 23a makes 
the account charge server 24 perfonn the charge processing of cost for the retrieval process via the network 1 . 

55 [0055] Further, in this specification, encoding means to convert the image data into data of the foreground component 
image, the background component image, the motion vector, the positional infonnation, the amount of motion blur and 
the mixture ratio infonnation, and the data thereof is referred to as encoded data. 
[0056] FIG. 27 is a block diagram showing the separation sen/er 11 . 
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[0057] Further, it does not matter whether each function of the separation server 1 1 is implemented by hardware or 
software. That is, each block diagram of the present invention may be considered as a blocl( diagram of hardware or 
as a functional block diagram of software. 

[0058] The input image supplied to the separation server 11 is supplied to an object extracting unit 101 , an area 
specifying unit 103, a mixture ratio calculator 104 and a foreground/background separator 105. 
[0059] The object extracting unit 1 01 roughly extracts an image object con-esponding to the foreground object of the 
Input image and supplies the extracted image object to the motion detector 1 02. The object extracting unit 1 01 detects, 
for example, the outline of the image object corresponding to the foreground object of the input image and roughly 
extracts the image object corresponding to the foreground object. 

[0060] The object extracting unit 101 roughly extracts an image object corresponding to the background object of 
the input image and supplies the extracted image object to the motion detector 102. The object extracting unit 101 
roughly extracts the image object corresponding to the background object on the basis of, for example, the difference 
between the input image and the image object corresponding to the extracted foreground object. 
[0061] Further, tor example, the object extracting unit 101 may roughly extract the Image object corresponding to 
the foreground object and the image object corresponding to the background object on the basis of the difference 
between the background image stored in an internally provided background memory and the input image. 
[0062] The motion detector 102 calculates a motion vector for a roughly extracted Image object con-esponding to 
the foreground object by using techniques such as a block matching method, an equally-dividing method, a phase 
correlating method or a pixel recursive method to supply the calculated motion vector and the positional infonnation 
of the motion vector (information specifying positions of pixels corresponding to the motion vector) to the area specifying 
unit 1 03 and the motion blur adjusting unit 1 06. The motion vector output from the motion detector 1 02 includes infor- 
mation con-esponding to the amount of movement v. 

[0063] Further, for example, the motion detector 1 02 may output the motion vector of each image object to the motion 
blur adjusting unit 106 along with the pixel positional Information specifying pixels of image object. 
[0064] The amount of movement v which indicates the positional variation of the image corresponding to the moving 
object Is expressed In units of pixel interval. For example, when the object image corresponding to the foreground is 
moved and displayed at a position that Is separated by 4 pixels in a frame following a reference frame, the amount of 
movement v of the object image corresponding to the foreground is 4. 

[0065] Further, the object extracting unit 1 01 and the motion detector 102 are required for adjusting the amount of 
motion blur con-esponding to the moving object. 

[0066] The area specifying unit 1 03 specifies the foreground area, the background area or the mixed area for each 
pixel of the input Image and supplies infonnation indicating the foreground area, the background area or the mixed 
area for each pixel, to the mixture ratio calculator 104, the foreground/background separator 105 and the motion blur 

adjusting unit 106. 

[0067] The mixture ratio calculator 104 calculates a mixture ratio corresponding to pixels In the mixed area on the 
basis of the Input Image and the area Information supplied from the area specifying unit 1 03 and supplies the calculated 
mixture ratio to the foreground/background separator 105. 

[0068] The mixture ratio a is a value indicating a ratio of image component (hereinafter, referred to as background 
component) corresponding to the background object In pixel values as shown in equation (3) to be described later. 
[0069] The foreground/background separator 105 separates the Input image into a foreground component image 
having only the image component (hereinafter, refen-ed to as foreground component) corresponding to the foreground 
object and a background component image having only the background component on the basis of the area infonmation 
supplied from the area specifying unit 103 and the mixture ratio a supplied from the mixture ratio calculator 104 to 
supply the foreground component image to the motion blur adjusting unit 106 and the selector 107. Further, It may be 
considered that the separated foreground component image is the final output. Compared to the conventional methods 
where only the foreground and background are specified and separated without considering the mixed area, more 
accurate foreground and background can be obtained. 

[0070] The motion blur adjusting unit 1 06 determines a unit of processing which specifies one or more pixels in the 
foreground component image, based on the amount of movement v known from the motion vector and the area infor- 
mation. The unit of processing Is a data specifying a group of pixels that Is a target of motion blur adjusting process. 
[0071 J The motion blur adjusting unit 106 adjusts the amount of motion blur in the foreground component image by 
removing the motion blur in the foreground component image, decreasing the amount of motion blur, increasing the 
amount of motion blur, etc. based on the amount of motion blur input to the separation server 11, the foreground 
component Image supplied from the foreground/background separator 105, the motion vector and the positional infor- 
mation thereof supplied from the motion detector 1 02 and the unit of processing, and outputs the foreground compon ent 
image of which the amount of motion blur is adjusted to the selector 107. The motion vector and the positional infor- 
mation thereof may not be used. 

[0072] Here, the motion blur means distortion included in an image corresponding to a moving object, generated by 
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the motion of object in real society which is a target of picking up an image and by picking up characteristics of sensors. 
[0073] The seiector 1 07 selects the foreground component image supplied from the foreground/background sepa- 
rator 105 or the foreground component image supplied from the motion blur adjusting unit 106 of whteh the amount of 
motion blur Is adjusted, based on. for example, a selection signal corresponding to a selection made by a user and 
5 then, outputs the selected foreground component image. 

[0074] Next, with reference to FIGs. 28 through 43. an input image supplied to the separation server 11 will be 
described. 

[0075] FIG. 28 is a diagram illustrating an Image taken by a sensor 76a. The sensor 76a comprises, for example, a 
CCD video camera, etc having a CCD (Charge Coupled Device) area sensor that is a solid state image pickup device. 

10 An object 112 con^esponding to a foreground in a real worid is moved, for example, horizontally from left to right In the 
drawing between the sensor and an object 111 corresponding to a background in a real worid. 
[0076] The sensor 76a picks up the image of the object 112 corresponding to the foreground along with the object 
corresponding to the background. The sensor 76a outputs the picked up in a frame unit. For example, the sensor 76a 
outputs an image having 30 frames per second. The exposure time of the sensor 76a may be set to 1/30 second. The 

IS exposure time means a time interval whteh starts at the point of converting the input light into charges and ends at the 
point when the Input light has converted into charges. Hereinafter, tiie exposure time is referred to as a shutter time. 
[0077] FIG. 29 Is a diagram Illustrating an arrangement of pixels. In FIG. 29, *A* through '1' indicate the respective 
pixels. The pixels are arranged on a plane con-esponding to an Image. A detection element corresponding to one pixel 
Is arranged on the sensor 76a. When the sensor 76a picks up an Image, the detection element outputs a pixel value 

20 corresponding to one pixel, constituting the image. For example, a position of the detection element in X direction 
corresponds to a horizontal posjtion on an Image and a position of the detection element in Y direction conresponds 
to a vertical position on an Image. 

[0078] As shown In FIG. 30, for example, the CCD detection element converts the input light into a charge for the 
time interval corresponding to the shutter time and stores the converted charge. The amount of charge is approximately 
25 proportional to the Intensity of input light and time Interval when light is input. The detection element adds the charge 
converted from the Input light to the charge already stored for the time interval con-esponding to the shutter time. That 
is, the detection element integrates the input light for the time interval corresponding to the shutter time to store the 
amount of charge corresponding to the integrated light. The detection element can be mentioned to have Integration 
effect with time. 

30 [0079] The charge stored in the detection element is converted Into a voltage value by means of a circuit not shown 
and the voltage value is converted Into a pixel value such as digital data, etc. to be output. Therefore, each pixel value 
output from the sensor 76a has a value projected to one-dlmenslonal space as a result of integrating some portion 

having spatial extension of an object corresponding to foreground or background with respect to the shutter time. 
[0080] The separation server 1 1 extracts any significant infomriation, for example, mixture ratio a, hidden in the output 
35 signal by means of storing operation of such sensor 76a. The separation server 11 adjusts the amount of distortion, 
for example, the amount of motion blur resulting from the mixture of the same image object of the foreground. Further, 
the separation server 11 adjusts the amount of distortion resulting from the mixture of Image object of tiie foreground 
and image object of the background. 

[0081] FIG. 31 A through 31 D are diagrams showing an image obtained by taking picture of an object corresponding 
40 to a moving foreground and an object corresponding to a stationary background. FIG. 31 A shows an Image obtained 
by taking picture of an object corresponding to a moving foreground and an object corresponding to a stationary back- 
ground. In the example shown in FIG. 31 A, the object con-esponding to the foreground moves horizontally from left to 
right with respect to the screen. 

[0082] FIG. 31 B is a model diagram in which the pixel values corresponding to one line of the image shown in FIG. 
<5 31 A are expanded In the time direction. The horizontal direction in FIG. 31 B con-esponds to the spatial direction X in 
FIG. 31 A. 

[0083] The pixel values of pixels in the background area comprise only the components of the background, that is. 
the components of the image con-esponding to the background object. The pixel values of pixels in the foreground 
area comprise only the components of the foreground, that is, the components of the image corresponding to the 

50 foreground object. 

[0084] The pixel values of pixels in the mixed area comprise the components of the background and the components 
of the foreground. Since the pixel values of pixels in the mixed area comprise the components of the background and 
the components of the foreground, the mixed area can be mentioned as a distortion area. Further, the mixed area is 
classified into a covered background area and an uncovered background area. 
55 [0085] The covered background area is a mixed area con-esponding to a front end portion in a direction In which the 
foreground object moves in respect to the foreground area and is an area in which the background components is 
hidden behind the foreground with time elapsing. 

[0086] In contrast, the uncovered background area is a mixed area corresponding to the rear end portion in a direction 
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in which the foreground object moves In respect to the foreground area and is an area In which the background com- 
ponents appear with time elapsing. 

[0087] As such, the image including the foreground area, the background area, the covered background area or the 
uncovered background area is Input to the area specifying unit 1 03, the mixture ratio cateulator 1 04 and the foreground/ 

5 background separator 105 as the Input image. 

[0088] FIG. 32 is a diagram illustrating the background area, the foreground area, the mixed area, the covered back- 
ground area, and the uncovered background area like above. When an image corresponds to the Image shown in FIG. 
31 A and 31 B, the background area Is a stationary portion, the foreground area is a moving portion, the'covered back- 
ground area of the mixed area is a portion being changed from the background to the foreground and the uncovered 

10 background area of the mixed area is a potion being changed from the foreground to the background. 

[0089] FIG. 33 Is a model diagram In which the pixel values of pixels being adjacent in a row are expanded in the 
time direction in the Image obtained by picking up an image of an object corresponding to the stationary foreground 
and an object corresponding to the stationary background. For example, as pixels being adjacent in a row, pixels being 
adjacent to a line of screen can be selected. 

IS [0090] Pixel values of F01 to F04 shown in FIG. 33 are pixel values of pixels corresponding to a stationary foreground, 
object. Pixel values of B01 to 804 shown In FIG. 33 are pixel values of pixels corresponding to a stationary background 
object. 

[0091] The vertical direction in FIG. 33 corresponds to time and time elapses from the upper side to the lower side 
in the drawing. The Position at the upper side of the rectangle In FIG. 33 corresponds to a time when the sensor 76a 
20 starts to convert of the input light to charges and the position at the lower side of the rectangle in FIG. 33 con-esponds 
to time when the sensor 76a finishes converting the input light Into charges. That is, the distance from the upper side 
to the lower side of the rectangle In FIG. 33 corresponds to a shutter time. 

[0092] Now, a case of when the shutter time equals to the frame Interval will be explained as an example. 
[0093] The horizontal direction in FIG. 33 corresponds to the spatial direction x shown in FIG. 31 A. More specifically, 
25 in the example shown in FIG. 33. the distance from the left side of rectangle indicated by "F01" to the right side of 
rectangle indicated by "804" In FIG. 33 corresponds to 8 times of a pixel pitch, that is, intervals of 8 pixels being 
continuous. 

[0094] When the object of foreground and the object of background are stationary, light to be Input to the sensor 76a 

is not varied for a time interval corresponding to a shutter time. 
30 [0095] Here, the time Interval con-esponding to the shutter time Is divided into two or more equivalent lengths. For 

example, if a virtual division number is 4, the model diagram shown In FIG. 33 can be shown as the model diagram 

shown in FIG. 9. The virtual division number Is established according to the amount of movement v for the shutter time 

of the object con-esponding to the foreground. For example. If the amount of movement v is defined as 4, the virtual 

division number is set at 4 and the time Interval con-esponding to the shutter time is divided Into 4 intervals. 
35 [0096] The uppermost row in the drawing coresponds to a first divided time interval after the shutter Is opened. The 

second row from the uppermost In the drawing corresponds to a second divided time interval afterthe shutter Is opened. 

The third row from the uppemiost In the drawing corresponds to a third divided time Interval afterthe shutter Is opened. 

The fourth row from the uppermost in the drawing corresponds to a fourth divided time interval after the shutter is 

opened. 

40 [0097] Now, the time Interval by which the shutter time is divided according to the amount of movement v is ref en-ed 
to as shutter timeA^. 

[0098] When the object conresponding to the f oregro und is stationary, since light input to the sensor 76a is not varied, 
the foreground component F0 1 /v has the same value the pixel value F01 divided by the virtual division n umber. Similarly, 
when the object corresponding to the foreground Is stationary, the foreground component F02N has the same value 
45 as the pixel value F02 divided by the virtual division number, the foreground component f03N has the same value as 
the pixel value F03 divided by the virtual division number and the foreground component F04A^ has the same value 
as the pixel value F04 divided by the virtual division number. 

[0099] When the object corresponding to the background is stationary, since light Input to the sensor 76a is not 
varied, the background component BOIA^ has the same value as the pixel value 801 divided by the virtual division 
50 number. Similarly, when the object con-esponding to the background is stationary, the background component B02yv 
has the same value as the pixel value 802 divided by the virtual division number, the background component 803A^ 
has the same value as the pixel value 803 divided by the virtual division number and the background component B04A^ 
has the same value as the pixel value 804 divided by the virtual division number. 

[0100] That is, when an object corresponding to the foreground is stationary, since light input to the sensor 76a is 
55 not varied for the time interval corresponding to the shutter time, the foreground component fO^N conresponding to a 
first shutter time/v after the shutter is opened, the foreground component F01/v corresponding to a second shutter 
XlmeN after the shutter is opened, the foreground component FO^N corresponding to a third shutter tImeA/ after the 
shutter is opened and the foreground component FOI/v corresponding to a fourth shutter tIme/v after the shutter is 
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opened all have the same value. F02/v to F04/v have also the same relation as FOI/v. 

[01 01 ] When an object corresponding to the background is stationary, since light input to the sensor 76a is not varied 
for the time interval corresponding to the shutter time, the background component BOIA^ corresponding to a first shutter 
time/v after the shutter is opened, the background component BO^N con-esponding to a second shutter time/v after 
5 the shutter is opened, the background component B01A^ corresponding to a third shutter timeA^ after the shutter Is 
opened and the background component BOI/v corresponding to a fourth shutter timeA/ after the shutter Is opened alt 
have the same value. B02/v to B04/V have the same value as 801 N. 

[0102] Next, a case of when an object corresponding to the foreground is moving and the object corresponding to 
the background Is stationary will be explained. 

10 [0103] FIG. 35 is a model diagram in which when an object corresponding to the foreground is moving toward the 
right side in the drawing, the pixel values of pixels in a line including the covered background area are developed in 
the time direction. In FIG. 35, the amount of movement v of the foreground is 4. Since one frame corresponds to a 
short time, it can be supposed that the object corresponding to the foreground is a rigid body and is moving at a uniform 
speed. In FIG. 35, an image of the object corresponding to the foreground is moving to be displayed at position displaced 

15 by 4 pixels in a frame following a reference frame. 

[0104] In FIG. 35, the pixels from the leftmost to the fourth from the leftmost belong to the foreground. In FIG. 35, 
the pixels from the fifth from the leftmost to the seventh from the leftmost belong to the mixed area that Is the covered 
background area. In FIG. 35, the rightmost pixel belongs to the background area. 

[0105] Since the object con-esponding to the foreground is moving to cover the object corresponding to the back- 
20 ground with time elapsing, the components included in the pixel values of pixels belonging to the covered background 
area Is changed from the background components to the foreground components at certain time point of the time 
Interval corresponding to the shutter time. 

[0106] For example, the pixel values M of which edge is a bold line in FIG. 35 is expressed as equation (1 ). 

|yrt=B02Af+B02A^+F07/v+F06A^ (1) 

[0107] For example, since the fifth pixel from the leftmost includes the background component corresponding to a 
first shutter time/v and the foreground component corresponding to a third shutter time/v, the mixture ratio a of the fifth 

30 pixel from the leftmost is 1 /4. Since the sixth pixel from the leftmost Includes the background component corresponding 
to a second shutter time/v and the foreground component corresponding to the second shutter time/v, the mixture ratio 
a of the sixth pixel from the leftmost is 1 /2. Since the seventh pixel from the leftmost includes the background component 
corresponding to the third shutter time/v and the foreground component con-esponding to the first shutter time/v, the 
mixture ratio a of the seventh pixel from the leftmost is 3/4. 

35 [0108] Since It can be supposed that the object corresponding to the foreground is a rigid body and the foreground 
image is moving at an unifonn speed to be displayed at a position displaced to the right by 4 pixels in a next frame, 
for example, the foreground component F07/V, of the fourth pixel from the leftmost in FIG. 35, corresponding to the 
first shutter time/v after the shutter is opened is the same as the foreground component of the fifth pixel from the leftmost 
In FIG. 35, corresponding to the second shutter time/v after the shutter is opened. Similarly, the foreground component 

40 F07/V is the same as the foreground component of the sixth pixel from the leftmost in FIG. 35 corresponding to the 
third shutter time/v after the shutter is opened and the foreground component of the seventh pixel from the leftmost in 
FIG. 35 corresponding to the fourth shutter time/v after the shutter is opened, respectively 

[0109] Since it can be supposed that the object coresponding to the foreground Is a rigid body and the foreground 
image is moving at an unifonn speed to be displayed at a position displaced to the right by 4 pixels in a next frame, 

45 for example, the foreground component F06/v of the third pixel from the leftmost In FIG. 35 corresponding to the first 
shutter tImeA/ after the shutter is opened Is the same as the foreground component of the fourth pixel from the leftmost 
in FIG. 35 corresponding to the second shutter time/v afterthe shutter is opened. Similariy, the foreground component 
F06/V is the same as the foreground component of the fifth pixel from the leftmost in FIG. 35 corresponding to the third 
shutter time/v after the shutter is opened and the foreground component of the sixth pixel from the leftmost In FIG. 35 

50 corresponding to the fourth shutter time/v after the shutter is opened, respectively. 

[01 10] Since It can be supposed that the object con-esponding to the foreground is a rigid body and the foreground 
image is moving at an unifonn speed to be displayed at a position displaced to the right by 4 pixels in a next frame, 
for example, the foreground component F05/V of the second pixel from the leftmost in FIG. 35 con-esponding to the 
first shutter time/v after the shutter is opened Is the same as the foreground component of the third pixel from the 

S5 leftmost in FIG. 35 corresponding to the second shutter time/v after the shutter is opened. Similariy, the foreground 
component F05/v is the same as the foreground component of the fourth pixel from the leftmost in FIG. 35 corresponding 
to the third shutter time/v after the shutter Is opened and the foreground component of the fifth pixel from the leftmost 
in FIG. 35 corresponding to the fourth shutter time/v after the shutter is opened, respectively. 
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[0111] Since it can be supposed that the object con^esponding to the foreground is a rigid body and the foreground 
image Is moving at an uniform speed to be displayed at a position displaced to the right by 4 pixels in a next frame, 
for example, the foreground component F04/v of the leftmost pixel in FIG. 35 corresponding to the first shutter time/v 
after the shutter is opened is the same as the foreground component of the second pixel from the leftmost In FIG. 35 
5 corresponding to the second shutter time/v after the shutter is opened. Similariy, the foreground component F04/v is 
the same as the foreground component of the third pixel from the leftmost in FIG. 35 con-esponding to the third shutter 
tIme/v after the shutter is opened and the foreground component of the fourth pixel from the leftmost In FIG. 35 corre- 
sponding to the fourth shutter time/v after the shutter is opened, respectively. 

[0112] The foreground area corresponding to a moving object may be mentioned as a distortion area because it 

10 includes the motion blur as described by the above. 

[0113] FIG. 36 is a model diagram In which when the foreground is moving toward the right side In the drawing, the 
pixel values of pixels In a line Including the uncovered background area are expanded In the time direction. In FIG. 36, 
the amount of movement v of the foreground is 4. Since one frame corresponds to a short time, It can be supposed 
that an object corresponding to the foreground is a rigid body and is moving at a unifonn speed. In FIG. 36, an image 

15 of the object corresponding to the foreground is moving toward the right side by 4 pixels in a next frame. 

[01 14] In FIG. 36, pixels from the leftmost to the fourth from the leftmost belong to the background area. In FIG. 36, 
pixels from the fifth from the leftmost to the seventh from the leftmost belong to the mixed area that is the uncovered 
background area. In FIG. 36, the rightmost pixel belongs to the background area. 

[0115] Since the object corresponding to the foreground covering the object corresponding to the background is 
20 moving to be displaced from the object coresponding to the background with time elapsing, the components included 
In the pixel values of pixels belonging to the uncovered background area is changed from the foreground components 
to the background components at certain time point of the time interval con'esponding to the shutter time. 
[0116] For example, the pixel values M' of which edge Is a bold line in FIG. 36 Is expressed as equation (2). 

M'=F02/V+F01/v+B26/v+B26/v (2) 

[0117] For example, since the fifth pixel from the leftmost Includes the background component corresponding to a 
third shutter time/v and the foreground component corresponding to a first shutter time/v, the mixture ratio a of the fifth 

30 pixel from the leftmost is 3/4. Since the sixth pixel from the leftmost Includes the background component corresponding 
to a second shutter time/v and the foreground component corresponding to the second shutter tlnrie/v, the mixture ratio 
a of the sixth pixel from the leftmost is 1 /2. Since the seventh pixel from the leftmost includes the background component 
corresponding to the first shutter time/v and the foreground component corresponding to the third shutter time/v, the 
mixture ratio a of the seventh pixel from the leftmost is 1/4. 

35 [0118] More generalizing combining equation (1) and equation (2), the pixel value M can also be expressed by 
equation (3). 



40 M^a^B + ^Fil^ (3) 

[0119] Here, a is a mixture ratio. B is a pixel value of the background and Fi/v is a foreground component. 
[0120] Since it can be supposed that the object corresponding to the foreground is a rigid body and is moving at an 

45 unifonn speed and the amount of movement v Is 4, for example, the foreground component FOI/v of the fifth pixel from 
the leftmost in FIG. 36 corresponding to the first shutter time/v after the shutter is opened is the same as the foreground 
component of the sixth pixel from the leftmost in FIG. 36 corresponding to the second shutter time/v after the shutter 
is opened. Similariy, the foreground component FOI/v Is the same as the foreground component of the seventh pixel 
from the leftmost in FIG. 36 corresponding to the third shutter tlme/v after the shutter Is opened and the foreground 

50 component of the eighth pixel from the leftmost in FIG. 36 corresponding to the fourth shutter tlme/v after the shutter 
is opened, respectively. 

[0121] Since it can be supposed that the object con-esponding to the foreground is a rigid body and is moving at an 
unlfonn speed and the virtual division number is 4, for example, the foreground component F02/V of the sixth pixel 
from the leftmost In FIG. 36 con'esponding to the first shutter time/v after the shutter Is opened Is the same as the 
55 foreground component of the seventh pixel from the leftmost In FIG. 36 corresponding to the second shutter time/v 
after the shutter is opened. Similarly, the foreground component F02/v is the same as the foreground component of 
the eighth pixel from the leftmost in FIG. 36 corresponding to the third shutter time/v after the shutter is opened. 
[0122] Since it can be supposed that the object corresponding to the foreground is a rigid body and is moving at an 
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uniform speed and the amount of movement v is 4, for example, the foreground component F03N of the seventh pixei 
from the leftmost in FiG. 36 con-esponding to the first shutter timeA^ after the shutter Is opened is the same as the 
foreground component of the eighth pixel from the leftmost In FIG. 36 corresponding to the second shutter timeAv after 

the shutter is opened. 

5 [0123] In the explanations of FIG. 34 to 36, it has been explained that the virtual division number is 4, however the 
virtual division number corresponds to the amount of movement v. The amount of movement v usually con-esponds to 
the moving speed of an object corresponding to the foreground. For example, when an object corresponding to the 
foreground is moving to be displayed at position 4 pixels to the right in a frame following a reference frame, the amount 
of movement v is defined as 4. in accordance to the amount of movement v, the virtual division number is set at 4. 

10 Similarly, for example, when an object corresponding to the foreground is moving to be displayed at a position 6 pixels 
to the right in a frame following a reference frame, the amount of movement v is defined as 6 and the virtual division 
number is set at 6. 

[0124] FIGS. 37 and 38 show a relationship between the foreground area, the background area, the mixed area 
having the covered background area and the uncovered background area and the foreground component or the back- 
15 ground component corresponding to the divided shutter time. 

[0125] FiG. 37 shows an example in which the pixels of the foreground area, the background area and the mixed 
area are extracted from an image including the foreground corresponding to the object moving in front of the stationary 
background. In an example shown in FIG. 37, an object A corresponding to the foreground is moving horizontally with 
respect to the screen. 

20 [0126] A frame #n+1 is a frame next to a frame #n and a frame #n+2 is a frame next to the frame #n+1 . 

[0127] A model In whteh the pixels of the foreground area, the background area and the mixed area extracted from 
one of the frame #n to the frame #n+2 and the pixel values of the extracted pixels are expanded in the time direction 
with the amount of movement v being 4 is shown in FIG. 38. 

[01 28] Since the object A corresponding to the foreground is moving, the pixel values of the foreground area comprise 
25 4 different foreground components according to intervals of a shutter timeA/. For example, the leftmost pixel In the 
foreground area shown in FIG. 38 comprises FOIA^, F02A^, F03/v and F04Af. That Is, the pixels of the foreground area 
include the motion blur. 

[0129] Since an object corresponding to the background is stationary, the light con-esponding to the background 
input to the sensor 76a for the time interval corresponding to the shutter time does not change. In this case, the pixel 

30 values of the background area do not include the motion blur. 

[0130] The pixel values of pixels belonging to the mixed area containing the covered background area or the uncov- 
ered background area is comprised the foreground components and the background components. 
[0131] Next, a model will be explained that when an image corresponding to an object is moving, the pixel values of 
pixels are adjacent in one row in a plurality of frames and are at the same positions in the frames are expanded in the 

35 time direction. For example, when the image con-esponding to the object is moving horizontally with respect to the 
screen, the pixels being in one line on the screen can be selected as pixels being adjacent in one row. 
[0132] FIG. 39 is a model diagram in which the pixel values of pixels are expanded in the time direction, wherein the 
pixel values of pixels whteh are adjacent in one row in 3 frames of an image obtained by picking up an image of an 
object corresponding to the stationary background and are at the same positions in the frames. A frame #n Is a frame 

40 following a frame #n-1 and a frame #n+1 is a frame following the frame #n. Other frames are referred in the same 
manner. 

[0133] The pixel values of B01 to B12 shown in FIG. 39 are the pixel values corresponding to the object of the 
stationary background. Since the object con-esponding to the background is stationary, the pixel values of the corre- 
sponding pixels do not changed in the frame #n-1 to the frame # n+1 . For example, a pixel In the frame #n and a pixel 
45 in the frame #n+1 corresponding to the position of pixel having the pixel value of BOS in the frame #n-1 have the same 
pixel value of BOS, respectively. 

[0134] FIG. 40 is a model diagram in which the pixel values of pixels are expanded in the time direction, wherein the 
pixel values of pixels are adjacent in one row in 3 frames of an image obtained by picking up an image of an object 
corresponding to the foreground moving toward the right side in the drawing along with an object corresponding to the 
50 stationary background and are at the same positions in the frames. The model shown in FIG. 40 includes the covered 
background area. 

[0135] In FIG. 40, since it can be supposed that the object con-esponding to the foreground is a rigid body and is 
moving at an uniform speed and the foreground image is moving to be displayed displaced to the right by 4 pixels in 
a next frame, the amount of movement v of the foreground is defined as 4 and the virtual division number is set at 4. 
55 [0136] For example, the foreground component of the leftmost pixel of a frame #n-1 In FIG. 40 con-esponding to a 
first shutter XlmeN after the shutter is opened is F1 2/v, and the foreground component of the second pixel from the 
leftmost in FIG. 40 corresponding to a second shutter timeA^ after the shutter is opened is also F12A^. Similarly, the 
foreground component of the third pixel from the leftmost In FIG. 40 corresponding to a third shutter tImeAv after the 
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shutter is opened and the foreground component of the fourth pixel from the leftmost In FIG. 40 corresponding to a 
fourth shutter time/v after the shutter is opened are F12/v. 

[0137] The foreground component of the leftmost pixel of the frame #n-1 in FIG. 40 corresponding to the second 
shutter time/v after the shutter is opened is Fll/v, and the foreground component of the second pixel from the leftmost 
in FIG. 40 corresponding to the third shutter time/v after the shutter Is opened is aiso F11/v. The foreground component 
of the third pixel from the leftmost in FIG. 40 corresponding to the fourth shutter time/v after the shutter is opened is F1 1/v. 
(01 38] The foreground component of ttie leftmost pixel of the frame #n-1 in FIG. 40 conresponding to the third shutter 
time/v after the shutter is opened is F1 0/v, and the foreground component, of the second pixel from the leftmost In FIG. 
40 corresponding to the fourth shutter time/v after the shutter is opened is aiso FIO/v. The foreground component of 
the leftmost pixel of the frame #n-1 In FIG. 40 corresponding to the fourth shutter time/v after the shutter is opened is 
F09N. 

[0139] Since an object corresponding to the background is stationary, the background component of the second pixel 
from the leftmost of the frame #n-1 in FIG. 40 corresponding to the first shutter time/v after the shutter is opened is 
B01/V. The background components of the third pixel from the leftmost of the frame #n-1 In FIG. 40 con-espondlng to 
the first and the second shutter tIme/v after the shutter is opened are B02^. The background components of the fourth 
pixel from the leftmost of the frame #n-1 in FIG. 40 corresponding to the first to third shutter time/v after the shutter is 
opened are B03/v. 

[0140] In the frame #n-1 of FIG. 40, the leftmost pixel corresponds to the foreground area and the second to fourth 
pixels from the leftmost belong to the mixed area that is the covered background area. 

[0141] The fifth to twelfth pixels from the leftmost of the frame #n-1 in FIG. 40 belong to the background area and 
the pixel values thereof are B04 to 811 , respectively. 

[01 42] The first to fifth pixels from the leftmost of a frame #n In FIG. 40 belong to the foreground area. In the foreground 
area of the frame #n, the foreground component of the shutter tIme/v Is one of FOS/v to F1 2/v. 
[0143] Since it can be supposed that the object corresponding to the foreground is a rigid body and is moving at an 
uniform speed and the foreground Image Is moving to be displayed displaced to the right by 4 pixels In a next frame, 
the foreground component of the fifth pixel from the leftmost of the frame #n in FIG. 40 corresponding to the first shutter 
tIme/v after the shutter is opened is F12/v, and the foreground component of the sixth pixel from the leftmost In FIG. 
40 corresponding to the second shutter time/v after the shutter is opened is also F1 2/v. Similarly, the foreground com- 
ponent of the seventh pixel from the leftmost in FIG. 40 con-esponding to the third shutter time/v after the shutter is 
opened and the foreground component of the eighth pixel from the leftmost In FIG. 40 corresponding to the fourth 
shutter time/v after the shutter Is opened are F12/V. 

[0144] The foreground component of the fifth pixel from the leftmost of the frame #n In FIG. 40 corresponding to the 
second shutter tIme/v after the shutter Is opened Is FH/v, and the foreground component of the sixth pixel from the 
leftmost In FIG. 40 corresponding to the third shutter tIme/v after the shutter is opened Is also F11/v. The foreground 
component of the seventh pixel from the leftmost in FIG. 40 con-esponding to the fourth shutter time/v after the shutter 

is opened is F11/v. 

[0145] The foreground component of the fifth pixel from the leftmost of the frame #n In FIG. 40 corresponding to the 
th Ird shutter time/v after the shutter is opened is F1 0/v, and the foreground component of the sixth pixel from the leftmost 
in FIG. 40 corresponding to the fourth shutter time/v afterthe shutter is opened is aiso F1 0/v. The foreground component 
of the fifth pixel from the leftmost of the frame #n in FIG. 40 con-espondlng to the fourth shutter time/v afterthe shutter 
is opened is F09A^. 

[0146] Since the object corresponding to the background is stationary, the background component of the sixth pixel 
from the leftmost of the frame #n in FIG. 40 corresponding to the first shutter tIme/v after the shutter Is opened is B05/V, 
The background components of the seventh pixel from the leftmost of the frame #n in FIG. 40 con-esponding to the 
first and the second shutter time/v after the shutter is opened are B06/v. The background components of the eighth 
pixel from the leftmost of the frame #n in FIG. 40 corresponding to the first to third shutter time/v after the shutter Is 
opened are B07/v. 

[0147] In the frame #n In FIG. 40, the sixth to eighth pixels from the leftmost belong to the mixed area that is the 
covered background area. 

[0148] The ninth to twelfth pixels from the leftmost of the frame #n in FIG. 40 belong to the background area and the 
pixel values thereof are BOB to B11 , respectively. 

[0149] The first to ninth pixels from the leftmost of a frame #n+1 in FIG. 40 belong to the foreground area. In the 
foreground area of the frame #n+1 , the foreground component conesponding to the shutter time/v is one of F01/v to 
F12/V. 

[0150] Since It can be supposed that the object con-esponding to the foreground is a rigid body and is moving at an 
unifomi speed and the foreground image is moving to be displayed displaced to the right by 4 pixels in a next frame, 
the foreground component of the ninth pixel from the leftmost of the frame #n+1 in FIG. 40 corresponding to the first 
shutter time/v after the shutter is opened is F1 2/v, and the foreground component of the tenth pixel from the leftmost 
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In FIG. 40 corresponding to the second shutter time/v after the shutter is opened is also F12/v. The foreground com- 
ponent of the eleventh pixel from the leftmost in FIG. 40 corresponding to the third shutter tIme/v after the shutter is 
opened and the foreground component of the twelfth pixel from the leftmost in FIG. 40 corresponding to the fourth 
shutter time/v after the shutter is opened are F1 2/v. 
5 [0151] The foreground component of the ninth pixel from the leftmost of the frame #n+1 in FIG. 40 con-esponding to 
the second shutter time/v after the shutter is opened is F1 1/v, and the foreground component of the tenth pixel from 
the leftmost in FIG. 40 conresponding to the third shutter tmeN after the shutter is opened is also F1 1/v. The foreground 
component, of the eleventh pixel from the leftmost in FIG. 40, corresponding to the fourth shutter time/v after the shutter 
is opened is F11 /v. 

10 [0152] The foreground component of the ninth pixel from the leftmost of the frame #n+1 in FIG. 40 con-esponding to 
the third shutter time/v after the shutter is opened is FIO/v, and the foreground component of the tenth pixel from the 
leftmost in FIG. 40 corresponding to the fourth shutter time/v after the shutter is opened is also F1 0/v. The foreground 
component of the ninth pixel from the leftmost of the frame #n+1 in FIG. 40 corresponding to the fourth shutter time/v 
after the shutter is opened is F09/v. 

15 [01 53] Since the object corresponding to the background is stationary, the background component of the tenth pixel 
from the leftmost of the frame #n+1 in FIG. 40 corresponding to the first shutter time/v after the shutter is opened is 
B09/V. The background components of the eleventh pixel from the leftmost of the frame #n+1 In FIG. 40 corresponding 
to the first and the second shutter time/v after the shutter is opened are BIO/v. The background components of the 
twelfth pixel from the leftmost of the frame #n+1 in FIG. 40 con-esponding to the first to third shutter time/v after the 

20 shutter is opened are B1 1 /v. 

[0154] In the frame #n+1 of FIG, 40, the tenth to twelfth pixels from the leftmost belong to the mixed area that is the 
covered background area. 

[0155] FIG. 41 is a model diagram of an image obtained by extracting the foreground components from the pixel 
values shown in FIG. 40. 

25 [0156] FIG. 42 is a model diagram in which the pixel values of pixels are expanded in the time direction, wherein the 
pixel values of pixels are adjacent in one row in 3 frames of an Image obtained by picking up an image of an object 
corresponding to the foreground moving toward the right side in the drawing along with the stationary background and 
are at the same positions in the frames. In FIG. 42. the uncovered background area is Included. 
[0157] In FIG. 42, it can be supposed that the object corresponding to the foreground is a rigid body and is moving 

30 at a uniform speed. Since the object con-esponding to the foreground is moving to be displayed displaced to the right 
by 4 pixels in a next frame, the amount of movement v of the foreground is defined as 4. 

[0158] For example, the foreground component of the leftmost pixel of a frame #n-1 in FIG. 42 con-esponding to a 
first shutter time/v after the shutter is opened is F13/v, and the foreground component of the second pixel from the 
leftmost in FIG. 42 corresponding to a second shutter time/v after the shutter is opened is also F1 3/v. The foreground 
35 component of the third pixel from the leftmost in FIG. 42 corresponding to a third shutter time/v after the shutter is 
opened and the foreground component of the fourth pixel from the leftmost in FIG. 42 corresponding to a fourth shutter 
time/v after the shutter is opened are F13A^. 

[0159] The foreground component of the second pixel from the leftmost of the frame #n-1 in FIG. 42 corresponding 
to the first shutter time/v after the shutter is opened is F14/v, and the foreground component of the third pixel from the 
40 leftmost In FIG. 42 corresponding to the second shutter time/v after the shutter Is opened is also F14/v. The foreground 
component of the third pixel from the leftmost in FIG. 42 corresponding to the first shutter tIme/v after the shutter is 
opened Is F15/V. 

[0160] Since an object corresponding to the background is stationary, the background components of the leftmost 
pixel of the frame #n-1 in FIG. 42 con-esponding to the second to fourth shutter time/v after the shutter Is opened are 
45 B25/V. The background components of the second pixel from the leftmost of the frame #n-1 in FIG. 42 corresponding 
to the third and fourth shutter time/v after the shutter is opened are B26/V. The background component of the third pixel 
from the leftmost of the frame #n-1 In FIG. 42 corresponding to the fourth shutter time/v after the shutter is opened is 
B27/V. 

[0161] In the frame #n-1 of FIG. 42. the first, second and third pixels from the leftmost belong to the mixed area that 
50 Is the uncovered background area. 

[0162] The fourth to twelfth pixels from the leftmost in the frame #n-1 In FIG. 42 belong to the foreground. The 
foreground components of the frame are one of F1 3/v to F24/v. 

[0163] The first, second, third and fourth pixels from the leftmost in the frame #n In FIG. 42 belong to the background 
area and the pixel values thereof are B25 to B28, respectively. 
55 [0164] Since it can be supposed that the object corresponding to the foreground Is a rigid body and is moving at an 
unifonn speed and the foreground image is moving to be displayed displaced to the right by 4 pixels in a next frame, 
the foreground component of the fifth pixel from the leftmost of the frame #n in FIG. 42 corresponding to the first shutter 
tIme/v after the shutter is opened is F13/V, and the foreground component of the sixth pixel from the leftmost in FIG. 
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42 corresponding to the second shutter time/v after the shutter is opened is also F13/v. The foreground component of 
the seventh pixel fronn the leftmost in FIG. 42 corresponding to the third shutter time/v after the shutter is opened and 
the foreground component of the eighth pixel from the leftmost in FIG. 42 con^esponding to the fourth shutter tIme/V 
after the shutter Is opened are F1 3/v. 
5 [0165] The foreground component of the sixth pixel from the leftmost of the frame #n in FIG. 42 corresponding to 
the first shutter time/v after the shutter is opened Is F14/v, and the foreground component of the seventh pixel from 
the leftmost In FIG. 42 corresponding to the second shutter tIme/v after ttie shutter is opened is also F14/v. The fore- 
ground component of the eighth pixel from the leftmost in FIG. 42 con-esponding to the first shutter time/v after the 
shutter is opened is F15/v. 

10 [0166] Since the object corresponding to the background is stationary, the background components of the fifth pixel 
from the leftmost of the frame #n in FIG. 42 corresponding to the second to the fourth shutter time/v after the shutter 
is opened are B29A^. The background components of the sixth pixel from the leftmost of the frame #n in FIG. 42 
corresponding to the third and the fourth shutter tIme/v after the shutter is opened are B30/v. The background compo- 
nents of the seventh pixel from ttie leftmost of ttie frame #n in FIG. 42 con-esponding to the fourth shutter tIme/v after 

15 the shutter Is opened are B31/v. 

[0167] In the frame #n of FIG. 42, the fifth to seventh pixels from the leftmost belong to the mixed area that is the 
uncovered background area. 

[0168] The eighth to twelfth pixels from the leftmost of the frame #n in FIG. 42 belong to the foreground area. In the 
foreground area of the frame #n, the values corresponding to the shutter time/v are one of F1 3/v to F20/v. 
20 [0169] The leftmost to eighth pixels from the leftmost of a frame #n+1 in FIG. 42 belong to the background area, and 
the pixel values thereof are B25 to B32. respectively. 

[0170] Since it can be supposed that the object corresponding to the foreground Is a rigid body and is moving at an 
unifonn speed and the foreground image is moving to be displayed displaced to the right by 4 pixels in a next frame, 
the foreground component of the ninth pixel from the leftmost of the frame #n+1 in FIG. 42 corresponding to the first 

25 shutter time/v after the shutter is opened is F1 3/v, and the foreground component of the tenth pixel from the leftmost 
in FIG. 42 corresponding to a second shuttertime/v after the shutter is opened is also F1 3/v. The foreground component 
of the eleventh pixel from the leftmost in FIG. 42 corresponding to the third shutter time/v after the shutter is opened 
and the foreground component of the twelfth pixel from the leftmost In FIG. 42 corresponding to the fourth shutter ttme/ 
V after the shutter is opened are F13/V. 

30 [01 71 ] The foreground component of the tenth pixel from the leftmost of the frame #n+1 In FIG. 42 corresponding to 
the first shutter time/v after the shutter is opened is F14/v, and the foreground component of the eleventh pixel from 
the leftmost in FIG. 42 corresponding to the second shuttertime/v after the shutter Is opened is also F14/v. The fore- 
ground component of the tweltth pixel from the leftmost In FIG. 42, corresponding to the first shutter time/v after the 
shutter is opened Is F15/v. 

35 [0172] Since the object corresponding to the background is stationary, the background components of the ninth pixel 
from the leftmost of the frame #n+1 in FIG. 42 corresponding to the second to the fourth shutter time/v after the shutter 
is opened are B33/v. The background components of the tenth pixel from the tettmost of the frame #n+1 in FIG. 42 
corresponding to the third and the fourth shutter tIme/v after the shutter is opened are B34/v. The background compo- 
nent of the eleventh pixel from the leftmost of the frame #n+1 in FIG. 42 corresponding to the fourth shuttertime/v after 

40 the shutter is opened is B35/v. 

[0173] In the frame #n+1 of FIG. 42, the ninth to eleventh pixels from the leftmost belong to the mixed area that is 
the uncovered background area. 

[0174] The twelfth pixel from the leftmost of the frame #n+1 in FIG. 42 belongs to the foreground. In the foreground 
area of the frame #n+1 , the foreground components corresponding to the shutter tIme/v are one of F1 3/v to F1 6/v. 
45 [0175] FIG. 43 is a model diagram of an image obtained by extracting the foreground components from the pixel 

values shown In FIG. 42. 

[0176] Returning to FIG. 27, the area specifying unit 103 generates flags indicating which of the foreground area, 
the background area, the covered background area or the uncovered background area correspond to the respective 
pixels, using a plurality of pixel values, and then, supplies them to the mixture ratio calculator 1 04 and the motion blur 
50 adjusting unit 1 06 as the area Information. 

[0177] The mixture ratio calculator 104 calculates the mixture ratios a for each pixel included in the mixed area on 
the basis of the pixel values irom a plurality of frames and the area information, and supplies the calculated mixture 
ratio a to the foreground/background separator 1 05. 

[0178] The foreground/background separator 105 extracts the foreground component image having only the fore- 
55 ground components on the basis of the pixel values from a plurality of frames, the area infonnation and the mixture 
ratio a and supplies it to the motion blur adjusting unit 106. 

[0179] The motion blur adjusting unit 106 adjusts the amount of motion blur Included In the foreground component 
image on the basis of the foreground component Image supplied from the foreground/background separator 105, the 
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[0199] The area determining portion 203-2 supplies the stationary area detemninating flag to which 'I' or '0" is set 
as described by the above to the frame memory storing determinating flag 204. 

[0200] When the stationary/moving detenmination supplied from the stationary/moving detenmining portion 202-2 
indicates motion status and the static naryMioving detemiination supplied from the stationary/moving detemriining por- 

5 tion 202-3 indicates motion status, the area determining portion 203-2 determines that the pixel being a target of the 
area specification in the frame #n belongs to the moving area and sets "1" indicating that the pixel belongs to the 
moving area to the moving area detemninating flag corresponding to the pixel of which area is detenmlned. 
[0201] When the stationary/moving determination supplied from the stationary/moving determining portion 202-2 
indicates stationary status or the stationary/moving determination supplied from the stationary/moving determining 

10 portion 202-3 indicates stationary status, the area determining portion 203-2 detemiines that the pixel being a target 
of the area specification in the frame #n does not belong to the moving area and sets "0" indicating that the pixel does 
not belong to the moving area to the moving area detemninating flag corresponding to the pixel of which area Is deter- 
mined. 

[0202] The area detemriining portion 203-2 supplies the moving area detemilnating flag to which "1" or "0" is set as 

15 described by the above to the frame memory storing determinating flag 204. 

[0203] When the stationary/moving detenmination supplied from the stationary/moving detemiining portion 202-3 
Indicates motion status and the stationary/moving detemiination supplied from the stationary/moving detemnlning por- 
tion 202-4 indicates stationary status, the area determining portion 203-3 detemiines that the pixel being a target of 
the area specification In the frame #n belongs to the covered background area and sets "1" Indicating that the pixel 

20 belongs to the covered background area to the covered background area detenminating flag corresponding to the pixel 
of whk:h area is determined. 

[0204] When the stationary/moving determination supplied from the stationary/moving determining portion 202-3 
indicates stationary status or the stationary/moving detemnination supplied from the stationary/moving determining 
portion 202-4 Indicates motion status, the area determining portion 203-2 detennines that the pixel being a target of 
25 the area specification in the frame #n does not belong to the covered background area and sets "0" Indicating that the 
pixel does not belong to the covered background area to the covered background area determinating flag corresponding 
to the pixel of which area is determined. 

[0205] The area determining portion 203-3 supplies the covered background area detemiinatlng flag to which *1 " or 
"0" is set as described by the above to the frame memory storing determinating flag 204. 

30 [0206] The frame memory storing detemilnating flag 204 stores the uncovered background area determinating flag 
supplied from the area determining portion 203-1, the stationary area detenninating flag supplied from the are deter- 
mining portion 203-2, the moving area detemilnating flag supplied from the area detemnlning portion 203-2 and the 
covered background area detenninating flag supplied from the area detennining portion 203-3, respectively 
[0207] The frame memory storing determinating flag 204 supplies the uncovered background area determinating 

35 flag, the stationary area detemninating flag, the moving area determinating flag and the covered background area 
detemninating flag stored therein to a synthesizer 205. The synthesizer 205 generates the area Infomnation indicating 
that each pixel belongs to any of the uncovered background area, the stationary area, the moving area and the covered 
background area on the basis of the uncovered background area determinating flag, the stationary area detenninating 
flag, the moving area detenninating flag and the covered background area detenninating flag supplied from the frame 

40 memory storing determinating flag 204 and then supplies it to the frame memory storing detemninating flag 206. 

[0208] The frame memory storing determinating flag 206 stores the area information supplied from the synthesizer 
205 and outputs the stored area infomnation. 

[0209] Next, referring to FIGs. 46 to 50, an example of processing by the area specifying unit 103 will be explained. 
[0210] When the object corresponding to the foreground is moving, the position on the screen of the image corre- 
45 spending to the object varies every frame. As shown in FIG, 46, the image con-espondlng to the object located at a 
position indicated by Yn(x, y) in the frame #n is located at Yn+1 (x, y) In the frame #n+1 . 

[021 1] FIG. 22 is a model diagram in which the pixel values of pixels being adjacent in a row in the moving direction 
of the image corresponding to the foreground object are expanded In the time direction. For example, when the moving 
direction of the Image corresponding to the foreground object Is horizontal on the screen, the model diagram in FIG. 
50 47 shows a model in which the pixel values of pixels being adjacent In one line are developed In the time direction. 
[0212] In FIG. 47, the lines in the frame #n are the same as those In the frame #n+1 . 

[021 3] The foreground components corresponding to the object included in the second to the thirteenth pixels from 
the leftmost in the frame #n are included in the sixth to the seventeenth pixels from the leftmost in the frame #n+1 . 
[0214] In the frame #n, pixels belonging to the covered background area are the eleventh to thirteenth pixels from 
55 the leftmost and pixels belonging to the uncovered background area are the second to fourth pixels from the leftmost. 
In the frame #n+1 , pixels belonging to the covered background area are the fifteenth to seventeenth pixels from the 
leftmost and pixels belonging to the uncovered background area are the sixth to eighth pixels from the leftmost. 
[0215] In the example shown In FIG, 47. since the foreground components Included In the frame #n Is moved by 4 
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background area. backaround is stationaiy, the pixel value of the fitteenth pixel from 

[0218] HerB,sincetheobiectcorrespond,ngtotheback^^^^ 

he leftmost In the frame #n-1 Is not changed from the pixel ^a^^^^^^^ J ^^^^^ the pixel 

#n-2. Similarly, the pixel value of the sixteen* P««' I'^"^^'; '^^^^^.^ ^ value of the seventeenth pixel from the 
value of the sixteenth pixel from the 'X^'reX^Ieom^^^^^^^^^ 

leftmostintheframc#n-1 is n°t'=*^«"9^^'^°'"f;P'!j'^^^^^^^ pixels belonging to the covered 

[0219] Thatis.sincepixeislntheframe#n-1andthet«m^^^^^ 

background area in the frame #n cornpnsed only the ^^2°";** ^JJ^^^ stationary/moving detem^inalion on 
Changed, the absolute 0^^^^^^^^^^ to Seis belonging to the mixed area In the frame #n 

Keiv=:r==^^^^^ 

«n-1 Therefore, the stationary/moving detemiining portion 202-3. 
pixetelnthecorrespondingframe#n-1isdeten.medtobem^^^^^ 

p)221] Like the above, when the station ao^/moymg '^^^^'^^f'^^T^^^ result Indicating stationary status 

Itatiolary/movlng detem,ining ponion ^JJ^^ ^^^^^^^^^^^^^ portion 203-3 detem,ines that 
is supplied from the stationary/moving detemiinmg portion ^o^ ^. 
the corresponding pixel belongs to the .^vered backgrou^^^ 

[0222] In the frame #n that is shown in FIG. 49 and in . ^^^^ the second to fourth pixels from 

l°of the foreground is 4. the pixels belonging to the "-/^^^^^^^^^^^ „ame #n.1 one frame behind 

the leftmost. Since the amount of -^"^^-^^^^ i*^" "tJ^^b^^^^ Trea. Further..the second to fourth pixels ,n 
include only the background components and b^^^^^^^^^^^^ 

theframe#n*2 further oneframebehlndincl^^^^^^^^^ 

[0223] Here, since the *ject corresponding to the backgro^ J ^^^^ ^^^^ 

he leftmost in the frame #n+2 is not f not changed from the pixel value 

#n.1. Similarly, the pixel value 0 the hirdp.^^^^^^^ 

of the third pixel from the leftmost ,r^ the '"^J^^J" J^j^^^^^ the leftmost in the frame #n+1 . 
#n+2 is not changed from the pixe value of the fourth P^^\^ ''^lll corresponding to the pixels belonging to the 
[0224] That is. since pixels in the frame ^^-^^^^^J"^^ ''^^Xound components and the pixel values thereof 
uncovered background area In the frame detemiinatton 
arenotchanged.theabsolutevalue«^thed^^^^^^^^^^ ^^^^^^^^ the mixed area In the frame 

rnrdS^renTs^o;^^^^^^^ 

SLv=or^^^^^^^ 

#n.1. Therefore, the stationary/moving de™^^^^^ 

pixels in the corresponding frame #n+1 IS detemiined to be mo^^^^^^ 

fo226] Uketheabove.whenthestat«na^/m^^^^^^^ 

r2rscrnS^=rofr^^ 

hat a pixel in the frame #n-2 located at the same posrtio J^^^^^^^^^^ ^3 the position in the image 

detem^lnation in the frame #n and a pixel .n the frame #n1 'JeSattonaP^ and when It Is detemilned that a pixel in the 
; of the pixel that is a target of detemi^aUon in the f rame f ^^^l^^^^^^^^,^^, that is a target of detem^ination in the 

re—o: i'n the frlme #n belongs to the covered background area. 
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the pixel in the frame #n-2 and the pixel In the frame #n-1 located at the same position and if it is determined to be 
stationary, the procedure proceeds to step S209. The stationary/moving determining portion 202-3 determines whether 
It is moving or not for the pixel in the frame #n-1 and the pixel In the frame #n located at the same position. 
[0242] In step S209, when it is determined to be moving for the pixel in the frame #n-1 and the pixel in the frame #n 
5 located at the same position, the procedure proceeds to step S210 and the area detemriining portion 203-3 sets T 
Indicating the pixel belongs to the covered background area to the covered background area determinating flag cor- 
responding to the pixel of whteh area is detemnlned. The area determining portion 203-3 supplies the covered back- 
ground area detenninating flag to the frame memory storing detenminating flag 204 and the procedure proceeds to 
stepS211. 

10 [0243] When it is detemnined to be moving for the pixel in the frame #n-2 and the pixel in the frame #n-1 position 
located at the same position in step S208 or when it is detemnined to be stationary for the pixel in the frame #n-1 and 
the pixel in the frame #n position located at the same position in step S209, since the pixel In the frame #n does not 
belong to the covered background area, the processing in step S210 is skipped and the procedure proceeds to step 
S211. 

15 [0244] In step S21 1 . the statlonaryAnoving determining portion 202-2 determines whether it is moving or not for the 
pixel In the frame #n and the pixel In the frame #n+1 located at the same position and If It Is determined to be moving, 
the procedure proceeds to step S212. The stationary/moving detemainlng portion 202-1 detennlnes whether it Is sta- 
tionary or not for the pixel in the frame #n+1 and the pixel in the frame #n+2 located at the same position. 
[0245] In step S212, when it is detennined to be stationary for the pixel in the frame #n+1 and the pixel In the frame 

20 #n+2 located at the same position, the procedure proceeds to step S21 3 and the area detemiining portion 203-1 sets 
"1" indteating the pixel belongs to the uncovered background area to the uncovered background area determinating 
flag corresponding to the pixel of which area Is determined. The area determining portion 203-1 supplies the uncovered 
background area determinating flag to the frame memory storing detenninating flag 204 and the procedure proceeds 
to step S214. 

25 [0246] When It is detemnined to be stationary for the pixel in the frame #n and the pixel in the frame #n+1 position 
located at the same position In step S211 or when it is detemnined to be moving for the pixel in the frame #n+1 and 
the pixel in the frame #n+2 position located at the same position in step S212, since the pixel in the frame #n does not 
belong to the uncovered background area, the processing In step S21 3 Is skipped and the procedure proceeds to step 
S214. 

30 [0247] In step S21 A, the area specifying unit 1 03 detemiines whether area specification has been performed for all 

the pixels in the frame #n or not and if it is determined that the area specification has not been performed for all the 
pixels in the frame #n. the procedure returns to step S202 and the area specifying processing Is repeatedfor otherplxels. 
[0248] In step S21 4, If it Is determined that the area specification has been perfomied for all the pixels in the frame 
#n, the procedure proceeds to step S21 5 and the synthesizer 205 generates the area infonnatlon indicating the mixed 
35 area on the basis of the uncovered background area determinating flag and the covered background area determinating 
flag stored in the frame memory storing determinating flag 204, generates the area infonnatlon Indicating that each 
pixel belongs to which of the uncovered background area, the stationary area, the moving area and the covered back- 
ground area and sets the generated area infonnation In the frame memory storing detenninating flag 206. Then, the 
processing ends. 

40 [0249] Like the above, the area specifying unit 1 03 can generate the area infonnatlon Indicating that the each pixel 
included in the frame belongs to the moving area, the stationary area, the uncovered background area or the covered 
background area. 

[0250] Further, the area specifying unit 103 may generate the area Infonnatlon corresponding to the mixed area by 
applying logical sum to the area information con'esponding to the uncovered background area and covered background 
45 area and may generate the area infonnation comprised of flags Indicating that the each pixel included in the frame 
belongs to the moving area, the stationary area or the mixed area. 

[0251] When the object con'esponding to the foreground has texture, the area specifying unit 1 03 can specify the 

moving area more accurately. 

[0252] The area specifying unit 1 03 can output the area infonnatlon indicating the moving area as the area infonnation 
50 Indicating the foreground area and the area Infonnatlon Indicating the stationary area as the area Infonnation indicating 
the background area. 

[0253] Further, although It has been explained that the object corresponding to the background is stationary, even 
when the Image corresponding to the background area includes motion, the area specifying processing described 
above can be applied. For example, when an image corresponding to the background area is moving constantly, the 
55 area specifying unit 1 03 shifts the whole image corresponding to this movement and processes It similarly to the case 
that the object corresponding to the background Is stationary. Further, when an image corresponding to the background 
area includes other movements locally, the area specifying unit 103 selects pixels conresponding to the movements 
and performs the aforementioned processing. 
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[0254] FIG. 54 is a block diagram illustrating another example of configuration of the area specifying unit 103. The 
area specifying unit 103 shown in FIG. 54 does not use the motion vector. A background image generator 301 generates 
the background image con'esponding to the Input image and supplies the generated background image to a binary 
object Image extracting portion 302. The background image generator 301 extracts, for example, an Image object 
con^esponding to the object of the background included in the input Image to generate the background image. 
[0255] An example of a model in whteh the pixel values of pixels being adjacent in a row in the moving direction of 
an image con-esponding to the foreground object are expanded in the time direction Is shown In FIG. 55. For example, 
when the moving direction of the image corresponding to the foreground object is horizontal on the screen, the model 
diagram shown in FIG. 55 shows a model in which the pixel values of pixels being adjacent in one line are expanded 
in the time direction. 

[0256] In FIG. 55. a line In a frame #n is the same as that in a frame #n-1 and a frame #n+1 . 
[0257] In the frame #n, the foreground components corresponding to the object Included In the sixth to seventeenth 
pixels from the leftmost are included in the second to thirteenth pixels from the leftmost in the frame #n-1 and in the 
tenth to twenty first pixels from the leftmost In the frame #n+1 . 

[0258] In the frame #n-1 , the pixels belonging to the covered background area are the eleventh to thirteenth pixels 
from the leftmost and the pixels belonging to the uncovered background area are the second to fourth pixels from the 
leftmost. In the frame #n, the pixels belonging to the covered background area are the fifteenth to seventeenth pixels 
from the leftmost pixels belonging to the uncovered background area are the sixth pixels to eighth pixels. In the frame 
#n+1 , the pixels belonging to the covered background area are the nineteenth to twenty first pixels from the leftmost 
and the pixels belonging to the uncovered background area are the tenth to twelfth pixels from the leftmost. 
[0259] In the frame #n-1 . the pixels belonging to the background area are the first pixel and the fourteenth to twenty 
first pixels from the leftmost. In the frame #n, the pixels belonging to the background area are the first to fifth pixels 
and the eighteenth to twenty first pixels from the leftmost. In frame #n+1 , the pixels belonging to the background area 
are the first to ninth pixels from the leftmost. 

[0260] An example of the background image con-esponding to the example of FIG. 55 which Is generated from 
background generator 301 is shown In FIG. 56. The background image Is comprised of pixels corresponding to the 
background object and does not include the image components corresponding to the foreground object. 
[0261] The binary object image extracting portion 302 generates the binary object image on the basis of the corre- 
lation between the background image and the input image and supplies the generated binary object image to a time 
change detector 303. 

[0262] FIG. 57 is a block diagram showing configuration of the binary object image extracting portion 302. A corre- 
lation value calculator 321 calculates the correlation between the input image and the background image supplied from 
the background image generator 301 and generates and supply the correlation value to a threshold value processor 
322. 

[0263] The correlation value calculator 321 applies equation (4) to, for example, 3x3 blocks in the background Image 
where X4 is centered as shown in FIG. 58A and 3x3 blocks in the input image, corresponding to the blocks in the 
background Image, where Y4 is centered as shown in FIG. 58B to calculate the con-elation value corresponding to Y4. 
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[0280] in step S302. the binary object image extracting portion 302 calculates the correlation value between the input 
image and the background image supplied from the background image generator 301 , for example, by using calcula- 
tions whteh were explained by referring to FIG. 58A and FIG. 58B. In step S303, the binary object image extracting 
portion 302 cak;utates the binary object Image from the correlation value and the threshold value thO, for example, by 
5 comparing the con-elation value with the threshold value thO. 

[0281 ] In step S304, the time change detector 303 performs the area detemriinlng processing to complete the process- 
ing. 

[0282] Referring to a flowchart of FIG. 65, the details of the area detemiining processing corresponding to step 5304 
will be explained. In step S321 , the area detennining portion 342 of the time change detector 303 determines whether 

^0 the indcated pixel in the frame #n stored in the frame memory 341 is 0 or not and if It Is detemnined that the indicated 
pixel in the frame #n is 0, the procedure proceeds to step S322. Then, the area detemnlnlng portion 342 detennlnes 
that the noticed pixel in the frame #n belongs to the background area to complete the processing. 
[0283] In step S321 , if it Is detenmined that the indicated pixel In the frame #n is 1 , the procedure proceeds to step 
S323. The area detemiining portion 342 of the time change detector 303 determines whether the indicated pixel in the 

IS frame #n stored in the frame memory 341 is 1 and the corresponding pixel in the frame #n-1 is 0 or not, and if It is 
determined that the indicated pixel in the frame #n Is 1 and the corresponding pixel in the frame #n-1 Is 0, the procedure 
proceeds to step S324. Then, the area determining portion 342 detennines that the indicated pixel in the frame #n 
belongs to the covered background area to finish the processing. 

[0284] in step S323, if rt is detemnlned that the indicated pixel in the frame #n Is 0 or the corresponding pixel in the 
^0 frame #n-1 is 1 , the procedure proceeds to step S325, The area detemiining portion 342 of the time change detector 
303 determines whether the indicated pixel in the frame #n stored in the frame memory 341 Is 1 and the corresponding 
pixel in the frame #n+1 is 0 or not, and if It is determined that the noticed pixel in the frame #n is 1 and the corresponding 
pixel In the frame #n-f 1 is 0, the procedure proceeds to step S326. Then, the area detemiining portion 342 detemnlnes 
that the indicated pixel In the frame #n belongs to the uncovered background area and finishes the processing. 
25 [0285] In step S325, if it Is detennined that the Indicated pixel In the frame #n Is 0 or the corresponding pixel In the 
frame #n+1 Is 1 , the procedure proceeds to step S327. The area determining portion 342 of the time change detector 
303 determines that the noticed pixel in the frame #n belongs to the foreground area and finishes the processing. 
[0286] Like the above, the area specifying unit 1 03 can specify which of the foreground area, the background area, 
the covered background area and the uncovered background area that the pixels of the input image belong on the 
30 basis of the correlation value between the Input image and the corresponding background image, and generates the 
area Infonmation corresponding to the specification result. 

[0287] FIG. 66 is a block diagram showing other configuration of the area specifying unit 103. The area specifying 
unit 1 03 shown in FIG. 66 uses the motion vector supplied from the motion detector 1 02 and the positional infonmation 
thereof. Portions equal to those in FIG. 64 are indicated by the same reference numerals and explanation thereof will 
35 be omitted. 

[0288] A robust processing portion 361 generates a binary object image of which robust processing is perfonned on 
the basis of the binary object image of N frames supplied from the binary object Image extracting portion 302 and 
outputs it to the time change detector 303. 

[0289] FIG. 67 Is a block diagram Illustrating a configuration of the robust processing portion 361 . A motion compen- 
<o sator 381 compensates for motion of the binary object image of N frames on the basis of the motion vector supplied 
from the motion detector 102 and the positional infomriation thereof, and outputs the binary object image of which 
motion is compensated to a switch 382. 

[0290] Referring to examples of FIG. 68 and FIG. 69, a motion compensation process of the motion compensator 
381 will be explained. For example, when an area in a frame #n is detennined, if the binary object image of a frame 
<5 #n-l, a frame #n and a frame #n+1 exemplified in FIG. 68 is Input, the motion compensator 381 compensates the 
binary object Image of the frame #n-1 and the binary object Image of the frame #n-t-1 , as exemplified In FIG. 69, on 
the basis of the motion vector supplied from the motion detector 102 and supplies the binary object image of which 
motion is compensated to the switch 382. 

[0291 ] The switch 382 outputs the binary object image of the first frame in which motion is compensated to the frame 
50 memory 383-1 and outputs the binary object image, of the second frame In which motion is compensated to the frame 
memory 383-2. Similarly, the switch 382 outputs each of the binary object images of the third to N-1 -th frames in which 
motion Is compensated to one of the frame memory 383-3 to 383-(N-1 ) and outputs the binary object image of the Nth 
frame in which motion is compensated to the frame memory 383-N. 

[0292] The frame memory 383-1 stores the binary object Image of the first frame in which motion is compensated 
55 and outputs the stored binary object image to the weighting portion 384-1 . The frame memory 383-2 stores the binary 
object Image of the second frame in which motion Is compensated and outputs the stored binary object image to the 
weighting portion 384-2. 

[0293] SImilariy, each of the frame memory 383-3 to 383-(N-1 ) stores one of the binary object images of the third to 
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(N-1 )th frames of which motion is compensated and outputs the stored binary object image to one of weighting portions 
384-3 to 384-(N-1). The frame memory 3B3-N stores the binary object image of Nth frame of which motion is compen- 
sated and outputs the stored binary object image to the weighting portion 384-N. 

[0294] The weighting portion 384-1 multiplies the binary object image of the first frame supplied from the frame 
memory 383-1 of which motion is compensated, by a predetennined weight w1 to supply it to the accumulator 385, 
The weighting portion 384-2 multiplies the pixel value of the binary object image of the second frame supplied from 
the frame memory 383-2 of which motion is compensated by a predetenmined weight w2 to supply {t to the accumulator 
385. 

[0295] Similarly, each of the weighting portion 384-3 to the weighting portion 384-(N-1) multiplies the pixel value of 
the binary object image of one of the third to (N-1 )th frames supplied from one of the frame memories 383-3 to 383-(N- 
1) of which motion is compensated, by one of weight w3 to weight w(N-1) to supply it to the accumulator 385. The 
weighting portion 384-N multiplies the pixel value of the binary object image of the Nth frame supplied from the frame 
memory 383-N of which motion is compensated; by a predetermined weight wN to supply It to the accumulator 385. 
[0296] The accumulator 385 accumulates the pixel values corresponding to the binary object images of the first to 
Nth frames respectively multiplied by one of the weights w1 to wN of which motions are compensated and generates 
the binary object image by comparing the accumulated pixel values with the predetemiined threshold value thO. 
[0297] Like the above, since the robust processing portion 361 generates the binary object images of which robust 
processing is perfomied from N binary object images and supplies it to the time change detector 303, even when noises 
are included in the Input image, the area specifying unit 103 of which the configuration is shown In FIG. 66 can perfomi 
the area specification more accurately than the case shown in FIG. 54. 

[0298] Next, referring to a flowchart of FIG, 70, the area specifying process of the area specifying unit 1 03 of which 
the configuration is shown in FIG. 66 will be explained. The steps of step S341 to step S343 are equal to those of step 
S301 to step S303 and thus, explanation thereof will be omitted. 
[0299] In step S344, the robust processing portion 361 perfonns the robust processing. 

[0300] In step S345, the time change detector 303 performs the area specifying process and finishes the procedure. 
Since the detailed process in step S345 is similar to the process explained by referring to the flowchart of FIG. 65, 
explanation thereof will be omitted. 

[0301] Next, refening to a flowchart of FIG. 71 , the details of the robust processing corresponding to the processing 
of step 8344 in FIG. 70 will be explained. In step S361 , a motion compensator 381 performs the motion compensating 
process for the input binary object image on the basis of the motion vector supplied from the motion detector 1 02 and 
the positional infomnation thereof. In step S362, one of the frame memories 383-1 to 383-N stores the motion com- 
pensated binary object image supplied through the switch 382. 

[0302] in step 8363, the robust processing portion 361 detemiines whether N binary object images are stored or not 
and if it Is determined that N binary object images are not stored, the procedure returns to step S361 and the motion 
compensating process for the binary object image and the storing process for the binary object image are repeated. 
[0303] In step S363, if it is determined that N binary object images are stored, the procedure proceeds to step S364. 
Each of the weighting portions 384-1 to 384-N multiplies each of the N binary object images by one of the weights w1 
to wN for weighting. 

[0304] In step S365, an accumulator 385 accumulates the weighted N binary object images. 

[0305] In step S366, the accumulator 385 generates a binary object Image from the accumulated Image, for example, 

by comparing to the predetemnined threshold value th1 and finishes the procedure. 

[0306] Like the above, the area specifying unit 1 03 of which the configuration is shown In FIG. 66 can generate the 
area infomiation on the basis of the binary object Image that Is robust processed. 

[0307] As described above, the area specifying unit 1 03 can generate the area infonmation indicating that each pixel 
included in the frame belongs to the moving area, the stationary area, the uncovered background area or the covered 
background area. 

[0308] FIG. 72 is a block diagram illustrating an example of a configuration of the mixture ratio calculator 104. An 
estimated mixture ratio processor 401 calculates the mixture ratio of each pixel by using the calculation corresponding 
to a model of the covered background area on the basis of the input Image and supplies the calculated estimated 
mixture ratio to the mixture ratio determining portion 403. 

[0309] An estimated mixture ratio processor 402 calculates the mixture ratio of each pixel by using the calculation 
corresponding to the model of the uncovered background area on the basis of the Input image and supplies the cal- 
culated estimated mixture ratio to the mixture ratio detennining portion 403. 

[0310] Since it is supposed that the object corresponding to the foreground is moving at a uniform speed for the 
shutter time, the mixture ratio a of pixel belonging to the mixed area has the following feature. That is, the mixture ratio 
a is linearly varied corresponding to the positional variation of a pixel, if the positional variation of a pixel is one- 
dimensional, the variation in mixture ratio a can be expressed by a linear line and if the positional variation of pixel is 
two-dimensional, the variation in mixture ratio a can be expressed by a plane. 
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[0311] Further, since time interval for one frame Is short, it can be supposed that the object corresponding to the 
foreground Is a rigid body and Is moving at a unifomi speed. 

[031 2] In this case, the slope of mixture ratio a is reciprocal ratio to the amount of movement v in the shutter time of 
the foreground. 

5 [031 3] An example of an Ideal mixture ratio a is shown In FIG. 73. The slope I of the Ideal mixture ratio a in the mixed 
area can be expressed as a reciprocal number of the amount of movement v. 

[0314] As shown In FIG. 73» the ideal mixture ratio a is 1 in the background area, is 0 In the foreground area and Is 
greater than 0 and less than 1 In the mixed area. 

[0315] In the example of FIG. 74, the pixel value COS of the seventh pixel from the leftmost in the frame #n can be 
10 expressed as equation (8) by using the pixel value P06 of the seventh pixel In the frame #n-1 . 



C06 = B06/V + BOdIv + FOl/v + F02/v 
= F06lv + P06lv + FOl/v + F02tv 
= 2iv^P06'^iFih (8) 

is=J 

20 [0316] In equation (8), the pixel value C06 is expressed as the pixel value M of pixel in the mixed area and the pixel 
value P06 is expressed as the pixel value B of pixel in the baclcground area. That Is, the pixel value M of pixel In the 
mixed area and the pixel value B of pixel In the background area can be expressed by equation (9) and equation (10), 
respectively. 
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M=C06 (9) 



B=P06 (10) 



[0317] 2/v in equation (8) con-esponds to the mixture ratio a. Since the amount of movement v is defined as 4, the 
mixture ratio a of the seventh pixel from the leftmost in the frame #n is 0.5. 

[0318] As described above, by considering the pixel value C in the Indicated frame #n as the pixel value in the mixed 
area and the pixel value P in the frame #n-1 ahead of the frame #n as the pixel value in the background area, the 
35 equation (3) expressing the mixture ratio a can be expressed by equation (11 ). 

C=a.P+f (11) 

40 f in equation (11) is sum Z|FiA^ of the foreground components Included in the indicated pixel. Variables Included 

in equation (1 1 ) are two of the mixture ratio a and the sum f of the foreground components. 
[0319] SImilariy, In the uncovered background area, the amount of movement v is 4 and the virtual division number 
In the time direction is 4. A model In which the pixel values are expanded in the time direction is shown in FIG. 75. 
[0320] in the uncovered background area, similar to the expressions of the aforementioned covered background 

45 area, by considering the pixel value C of pixel In the Indicated frame #n as the pixel value In the mixed area and the 
pixel value N In the frame #n-i-1 behind the frame #n as the pixel value in the background area, the equation (3) 
expressing the mixture ratio a can be expressed by equation (12). 



C=a.N+f (12) 



[0321] Further, although it has been explained by considering that the background object is stationary, even when 
the background object is moving, equation (8) to equation (12) can be applied by using the pixel value of pixel at a 
position corresponding to the amount of movement v of the background. For example, in FIG. 74, when the amount 
55 of movement v of the object corresponding to the background is 2, the virtual division number is 2 and when the object 
corresponding to the background is moving toward the right side in the drawing, the pixel value B of pixel In the back- 
ground area in equation (1 0) becomes the pixel value P04. 

[0322] Since each of equation (1 1 ) and equation (1 2) Includes two variables, the mixture ratio a cannot be obtained 
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by these equations. Here, since the Image has a strong correlation In space, the adjacent pixels are almost equal. 
[0323] Therefore, because the foreground components have a strong correlation in space, by changing the equations 
to derive the sum f of the foreground components from the front or next frame, the mixture ratio a is obtained. 
[03241 The pixel value Mc of the seventh pixel from the leftmost in the frame #n in FIG. 76 can be expressed by 
equation (13). 



Mc^4r^B06+f,Filv (13) 

[0325] 2N of the first term of right side of equation (13) con-esponds to the mixture ratio a. The second term of right 
side of equation (13) can be expressed by equation (14) using the pixel value in the next frame #n+1 . 

f,Filv^^^f,FHv (14) 

[0326] Here, using the spatial con-elation of the foreground components, equation (1 5) can be obtained. 

F=F05=F06=F07=F08=F09=F1 0=F1 1 =F1 2 (15) 
[0327] Equation (14) can be replaced with equation (16) by using equation (15). 

[0328] Consequently, p can be expressed by equation (17). 

p=2/4 (17) 

[0329] in general, If it is supposed that the foreground components corresponding to the mixed area as shown in 
equation (1 5) are equal, equation (1 8) can be established for all the pixels In the mixed area, in view of relation of the 
Interior division ratio. 

p=1-a (18) 
[0330] If the equation (1 8) is established, equation (11 ) can be obtained as shown in equation (1 9). 



i=:a-P+(i'-a)'J^FUv 



= a^P+(i'a)'N (^^) 
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[0372] In this equation (23), \ is an index In the spatial direction when the position of the indicated pixel is 0. 1 is 
slope of a linear line of the mixture ratio a. p is an intercept of the linear line of the mixture ratio a and is the mixture 
ratio a of the indicated pixel. In equation (23), the index i is known but the slope 1 and the intercept p are not known. 
[0373] The relationship between Index i, the slope 1 and the Intercept p is shown in FIG. 83. 
5 [0374] By approximating the mixture ratio a by equation (23), a plurality of different mixture ratios a for a plurality of 
pixels are expressed with two variables. In the example shown In FIG. 83. 5 mixture ratios for the fh^e pixels are 
expressed with two variables of the slope I and the intercept p. Further, In FIG. 83, the indicated pixel Is indicated by 
a white circle and the peripheral pixels are indicated by black circle. 

[0375] If the mixture ratio a is approximated in a plane shown in FIG. 84, considering movement v con-esponding to 
10 two directions, the horizontal direction and the vertical direction of the image, the equation (23) is extended into the 
plane and the mixture ratio a is expressed by equation (24). Further, in FIG. 84, the white circle indicates the indicated 
pixel. 
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a=JnrH-kq+p (24) 



[0376] In equation (24), J is an index in the horizontal direction when the position of the indicated pixel is 0 and k is 
an index in the vertical direction, m is a slope of the mixture ratio a plane in the horizontal direction and q is a slope of 
the mixture ratio a plane in the vertical direction, p is an intercept of the mixture ratio a plane. 
20 [0377] For example, in the frame #n shown in FIG. 74, equations (25) to (27) are established with respect to COS to 
C07, respectively. 



C05=a05-B05Af-ff05 (25) 
C06=a08-B06/v-ff06 (26) 
C07=a07.B07/v+f07 (27) 



[0378] if the foreground components are equal in the neighborhood, that is, F01 to F03 are equal and F01 to F03 is 
replaced with Fc, equation (28) can be established. 

f(x)={1.a(x)).Fc (28) 
[0379] In equation (28), x expresses a position of the spatial direction. 

[0380] If a (x) is replaced with equation (24), and equation (28) can be expressed by equation (29). 

f (x) = (l-(jia+kq+p)) - Fc 

• (-m-Fc)+)c- (-q-Fc) + ( (1-p) -Fc) 
=js+)ct+u (29) 

[0381] In equation (29), (-m-Fc), (-q-Fc) and (1-p)*Fc are replaced as shown In equations (30) to (32). 

so 

s=-m • Fc (30) 
55 t=-q . Fc (31) 

u=(1-p).Fc (32) 
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[0382] In equation (29). j is an index in the horizontal direction when position of the indicated pixel is 0 and k is an 
Index In the vertical direction. 

[0383] Like the above, since it is supposed that the object corresponding to the foreground is moving at a uniform 
speed for the shutter time and the components corresponding to the foreground are constant in the neighborhood, the 
sum of the foreground components is approximated by equation (29). 

[0384] Further, when the mixture ratio a is approximated by a linear line, the sum of the foreground components can 
be expressed by equation (33). 

f (X) = is+u (33) 

[0385] If substituting the mixture ratio a in equation (1 3) and the sum of the foreground components in equation (24) 
and equation (29), the pixel value M is expressed by equation (34). 

M= (jm+kq+p) • B+]s+kt+u 
= jB • m+kB • q+B • (>f j • s+k • t+u (34) 

[0386] In equation (34), the six unknown variables are the slope m of the mixture ratio a plane in the horizontal 
direction, the slope q of the mixture ratio a plane in the vertbal direction and the intercepts p, s, t and u of the mixture 
ratio a plane. 

[0387] The pixel value M or the pixel value B Is substituted in a normal equation shown in equation (34) corresponding 
to pixels In the neighborhood of the Indicated pixel and a plurality of nonnal equations in which the pixel value M or 
the pixel value B is substituted are solved by the least square method to calculate the mixture ratio a. 
[0388] For example, if the index j of the indicated pixel in the horizontal direction is 0, the index k in the vertical 
direction is 0 and the pixel value M or the pixel value B Is substituted In the nornial equation shown in equation (34) 



for 3x3 pixels adjacent to the Indicated pixel, equations (35) to (43) are obtained. 

(-■>) • B.i.-i • ^H-^) * ■ q+B.i .1 . p+{-1) ■ s+(-1) • t+u (35) 

mo,-i= W • ^0.-1 • • ^0.-1 " «?+Bo .1 . p+(0) . s+(-1) . t+u (36) 

M^i .1= (+1) . B^i ,1 . m+(-1) . B^i,.i . q+B^i .1 • p+(+1) . s+(-1) - t+u (37) 

M,i.o- (-1) • B.I 0 • nr)+(0) • B.^ q • q+B.^ q • P+M) • s+(0) • t+u (38) 

Mo,o=(0)-Bo.o'm+(0).Bo,o-q+Bo.o-P+(0)-s+(0)'t+u (39) 

M^1,0= (+1) • B+1.0 • ni+(0) * B^i,o • q+B^i,o * P+H) " ' (^0) 

M.1.41= (-1) ' B.1,^1 • m+(+1) . B.1 . q+B.^ • p+(.1) • s+(+1) - t+u (41) 

Mo,^,= (0) . Bo,+i . m+(+1) . B0.+1 • q+Bo.^1 • p+(0) • s+(+1) • t+u (42) 

M+i.4i= (+^) • • m+(+1) . B^i^^^ . q+B^i,+i • P+(+1) ' s+(+1) • t+u (43) 



[0389] Since the Index j of the indicated pixel In the horizontal direction is 0 and the index k in the vertical direction 
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is 0. the mixture ratio a of the Indicated pixel is equal to the value when j=:0 and k^O in equation (24). that is, the 
intercept p. 

[0390] Therefore, each value of the slope m In the horizontal direction, the slope q in the vertical direction and the 
Intercepts p, s> t and u Is calculated by using the least square method on the basis the nine equations (35) to (43) and 
5 then, the intercept p is output as the mixture ratio a. 

[0391] Next, a more specific procedure for calculating the mixture ratio a by using the least square method will be 
explained. 

[0392] If the index i and the index k are expressed by one Index x, the relationship of the index i, the Index k and the 
index x is expressed by equation (44). 

10 

x=0+1)-3+(k+1) (44) 

[0393] The slope m In the horizontal direction, the slope q in the vertical direction and the intercepts p, s, t, and u 
15 are expressed as variables wO, w1 , w2, w3, w4 and w5, respectively and jB, kB, 8, J, k and 1 are expressed as aO, a1 , 
a2, a3, a4 and a5, respectively. Considering error ex, equations (35) to (43) can be rewritten as equation (45). 

s 

20 Mx^^ayWy^ex (45) 



[0394] In equation (45), x is a value from integers 0 to 8. 
25 [0395] Equation (46) can be derived from the equation (45). 

s 

ex = Mx-^aywy (46) 

30 

[0396] Here, in order to use the least square method, the sum of squares of errors E is defined as shown by equation 
(47). 

35 

E=i:ex' (47) 

40 [0397] In order to minimize error, the partial differential of variable Wv and the sum of squares of errors E should be 
0. Here, v is one of Integers 0 to 5. Therefore, wy is obtained to satisfy equation (48). 



(48) 

50 

[0398] If equation (46) is substituted In equation (48), equation (49) Is obtained. 



45 



dWv "^xt-r* dWv 

a 

Xn0 
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8 5 8 

E (fl V 2 «y • y^y) ^^avMx (49) 

Xe>0 7sO XbO 



[0399] By apptying, for example, sweeping-out method such as Gauss-Jordan elimination method, to six equations 
obtained by substituting one of integers 0 to 5 in v of equation (49), wy is calcuiated. As described above, wO is the 
slope m in the horizontal direction, w1 is the slope In the vertical direction, w2 is the intercept p, w3 is s, w4 is t and w5 Is u. 
[0400] As described above, by applying the least square method to equations in which the pixel value M and the 
pixel value B are substituted, the slope m In the horizontal direction, the slope q in the vertical direction and the intercepts 
p, s, t and u can be obtained. 

[0401] In explanation of equations (35) to (43), the pixel value of pixel included in the mixed area is explained as M 
and the pixel value of pixel included in the baclcground area is explained as B. However, it is necessary that normal 
equations be established for each case that the indicated pixel is Included in the covered background area or in the 
uncovered background area. 

[0402] For example, when the mixture ratios a of pixels Included in the covered background area in the frame #n as 
shown in FIG. 74 are obtained, the pixel values C04 to COS of pixels in the frame #n and the pixel values P04 to P08 
of pixels In the frame #n-1 are substituted in the nomnal equations. 

[0403] When the mixture ratios a of pixels included in the uncovered background area In the frame #n as shown in 
FIG. 75 are obtained, the pixel values C28 to C32 of pixels in the frame #n and the pixel values N28 to N32 of pixels 
in the frame #n+1 are substituted in the nomrial equations. 

[0404] Further, for example, when the mixture ratios a of pixels included in the covered background area in the frame 
#n as shown In FIG. 85 are calculated, equations (50) to (58) can be established. The pixel value of pixel of which the 
mixture ratio a is calculated is Mc5. Further, in FIG. 85, white circles indicate the pixels in the background and black 



circles indicate the pixels in the mixed area, respectively. 

Mc1=: (-1) . Bc1 • m+(-1) . Bc1 • q+Bcl • p+(-1) . s+(-1) - t+u (60) 

IVIc2= (0) . Bc2 ■ m+(-1) • Bc2 - q+Bc2 • p+(0) • 8+(-1) • t+u (51) 

Mc3= (+1) . Bc3 • m+(-1) • Bc3 • q+Bc3 • p+(+1) • s+(-1) • t+u (52) 

Mc4= (-1 ) • Bc4 ■ m+(0) ■ Bc4 • q+Bc4 • p¥(-\ ) - s+(0) • t+u (53) 

I^c5= (0) . Bc5 ♦ m+(0) ♦ Bc5 • q+Bc5 • p+(0) • s+(0) • t+u (64) 

Mc6= (+1 ) • Bc6 . m+(0) • Bc6 . q+Bc6 - |>f (+1 ) • s+(0) • t+u (55) 

Mc7= (-1 ) . Bc7 . m+(+1 ) • Bc7 • q+Bc7 • p+(-1 ) ■ s+(+1 ) . t+u (66) 

Mc8= (0) . Bc8 . m+(+1) • Bc8 • q+Bc8 • p+(0) ■ s+(+1) • t+u (57) 

Mc9= (+1) . Bc9 • m+(+1) • Bc9 • q+Bc9 • p+(+1) • s+(+1) • t+u (68) 



[0405] When the mixture ratios a of pixels included In the covered background area in the frame #n are calculated, 
the pixel values Bc1 to Bc9 of pixels of the background area In the frame #n-1 corresponding to the pixels in the frame 
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#n are used in equations (50) to (58). 

[0406] When the mixture ratios a of pixels included in the uncovered background shown in FIG. 85 are calculated, 
equations (59) to (67) can be established. The pixel value of pixel of which the mixture ratio a is calculated is Mu5. 
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MuU (-1) • Bui • m+(-1) . Bui • q+Bul • p+(-1) • s+(-1) -t+u (59) 

IVIu2= (0) . Bu2 • m+(-1) - Bu2 • q+Bu2 • p+{0) • s+(-1) • t+u (60) 

Mu3= (+1) • Bu3 • m+(-1) - Bu3 - q+Bu3 • p+(+1) • s+(-1) • t+u (61) 

Mu4= (-1) . Bu4 - m+(0) ■ Bu4 • q+Bu4 • p+(-1) • s+{0) • t+u (62) 

Mu5= (0) . Hu5 . m+(0) • Bu5 • q+Bu5 • p+(0) - s+(0) • t+u (63) 

Mu6= (+1) • Bu6 . m+(0) - Bu6 • q+Bu6 • p+{+1) ■ s+(0) • t+u (64) 

Mu7= (-1) • Bu7 ■ m+(+1) • Bu7 • q+Bu7 • p+{-1) • s+(+1) • t+u (65) 

Mu8= (0) • Bu8 . m+(+1) • Bu8 • q+Bu8 • p+(0) • s+(+1) • t+u (66) 

IVIu9= (+1) . Bu9 . nfi+(+1) • Bu9 • q+Bu9 • p+(+1) • s+(+1) • t+u (67) 



[0407] When the mixture ratios a of pixels included In the uncovered background area in the frame #n are calculated, 

the pixel values Bui to Bu9 of pixels of the background area in the frame #n+1 corresponding to the pixels in the frame 

#n are used in equations (59) to (67). 
35 [0408] FIG. 86 Is a block diagram Illustrating configuration of the estimated mixture ratio processor 401 . The Image 

input to the estimated mixture ratio processor 401 is supplied to a delay circuit 501 and an adder 502. 

[0409] The delay circuit 501 delays the input image by one frame and supplies It to the adder 602. When the frame 

#n is input to the adder 502 as the input image, the delay circuit 501 supplies the frame #n-1 to the adder 502. 

[0410] The adder 602 sets the pixel values of pixels adjacent to pixel of which the mixture ratio a is calculated and 
40 the pixel values in the frame #n-1 in the noniial equations. For example, the adder 502 sets the pixel values Mcl to 

Mc9 and the pixel values Bel to Bc9 in the normal equations on the basis of equations (50) to (58). The adder 502 

supplies the normal equations in which the pixel values are set to the calculator 503. 

[041 1] The calculator 503 solves the nomial equations supplied from the adder 502 by using the elimination method, 
etc. to obtain the estimated mixture ratio and outputs the obtained estimated mixture ratio. 
45 [0412] Like the above, the estimated mixture ratio processor 401 can calculate the estimated mixture ratio on the 
basis of the input image and supply it to the mixture ratio determining portion 403. 

[041 3] Further, since the estimated mixture ratio processor 402 has the configuration similar to that of the estimated 
mixture ratio processor 401 , explanation thereof will be omitted. 

[0414] FIG. 87 shows an example of the estimated mixture ratio cateulated by the estimated mixture ratio processor 
50 401 , The estimated mixture ratio shown in FIG. 87 expresses the result calculated from equations generated in a unit 
of 7x7 pixels with respect to one line when the amount of movement v of the foreground corresponding to the object 
moving at a uniform speed Is 11 . 

[0415] It can be known that the estimated mixture ratio is varied almost linearly in the mixed area as shown in FIG. 86. 
[0416] Next, the mixture ratio estimating processing of the estimated mixture ratio processor 401 of which configu- 
55 ration Is shown in FIG. 86 by using a model corresponding to the covered background area will be explained referring 
to FIG. 88. 

[0417] In step S521 , the adder 502 sets the pixel values included in the input Image and the pixel values included 
in the Image supplied from the delay circuit 501 In the normal equations corresponding to the model of the covered 
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\SSr"^Z^SBZ>. the estimated mixture ratio processor 401 determines whether setting tor the ta^et pixete is 
fintehed or r,ot and If It is detemiined that the setting for the target pixels is not finished, the procedure returns to step 
S521 and repeats the setting processing of the pixel values in the nomial equations. , „^ ,k» ««v^,.rB 

104191 In step S522. If It Is detemiined that the setting the pixel value for the target pixels is finished, the procedure 
proceeds to step S523andthecalculator173 calculates the estimated mixture ratio on the basis of thenomiale^^^^^ 

in which the pixel values are set and outputs the obtained estimated mixture ratio. 

[0420] Like the above, the estimated mixture ratio processor 401 of which configuration Is shown in FIG. 86 can 

calculate the estimated mixture ratio on the basis of the Input Image. »„„,^h h=^i,nr«Mnri arpa is 

[0421] The mixture ratio estimating process using the model corresponding to the ^''^"^'^ ^^^'^l"^''^^ 
Lilarto the process usingthenomialequationscorresponding to the model of theunooveredbackgroundareashown 

in a flowchart of FIG, 88 and thus, explanation thereof will be omitted. .... , ^. =„on 

[0422] Further, although it has been explained that the object corresponding to the S^^^^ 
IhenLlmagecorrespondlngto the backgroundarea includes movement, the aforementlon«lm^^^^^^^ 

processing can be applied. For example, when the Image coirespondlng to the background is moving constantMhe 
eSSmixtureraSoprocessor401 shitlsthewhole image accordlngtothemovements and then.processessl^^^^ 

to the case that the object con-esponding to the background Is stationary. o«,i,T,ptPri mixture 

Further, when the image corresponding to the background area includes other local movements.theest,ma^^^^^ 

ratio processing portion 401 selects the pixels corresponding to the movements as the pixels corresponding to pixels 
belonging to the mixed area and then, perfomis the aforementioned processing. „, . • „ 

[0423] Like the above, the mixture ratio calculator 102 can calculate the mixture ratio a that is a feature amount 
Corresponding to each pixel on the basis of the area infomiatlon supplied from the area specifying unit 101 and the 

ISrS using the mixture ratio a. while the Infomiatlon of motion blur Included in the '-"^g^ "''[^^"f^^^ 
moving object Is left. It Is possible to separate the foreground components and the background components included 

[SSyTrttrif an Image is synthesized on the basis ofthemixtureratioa. it ispossibletogenera^^^^^^ 

the colected motion blur adapted to the speed of the object moving to correspond to ^^^^ jL^^f 
r04261 Next the foreqroundA)ackground separator 105 will be explained. FIG. 89 is a block diagram illustrating an 
exlm % Of a cXX'f the f o'^ground/background separator 1 05. The Input Image supplied to ^^J^>o^^^->;^^ 
backaround separrtor 105 Is supplied to a separating portion 601 . the switch 602 and the switch 604. The area infor- 

matSS t^^^^^^^ arSa and the area infomiatlon '"^ ^eraTnfo^^^^^^^ 

supplied from the area specifying unni03, are supplied to the separating portion 6m.T^^^^^ 

the foreground area is supplied to the switch 602. The area Infomiatlon Indtoating the background area Is supplied to 

15S"'^^e mixture ratio a supplied from the mixture ratio calculator 1 04 Is supplied to the separating porton Ml^ 
042?] The separating portton 601 separates the foreground components from the input image to supply the sepa^ 
aTed Lground components to the synthesizer 603 and separates the 
osupJytheseparatedbackgroundcLponentstothesymhes^ 

the covered background area, the area Infomiatlon Indicating the uncovered background area and ttie m^^^rev^^o a^ 
04297 When the pixels corresponding to the foreground are Input, the switch 602 •« c^«* 2 o 2 arJa 
pixels corresponding to the foreground included in the Input Image to the synthesizer 603. on the basis of the area 

infomiatlon Indicating the foreground area. ■ ^ „„h ei.nniiae nniutha 

[0430] When the pixels corresponding to the background are input, the swrtch 604 dosed and suPPj'; ""JyJJ; 
pixels corresponding to the background Included In the input image to the synthesizer 605. on the basis of the area 

infomiation indicating the background area. . ,.u„,„„„,„,,„rt^„mnnnpnts 

[0431] Thesyntheslzer603syntheslzestheforegn,undcomponentlmageonthebasisoftheoregra^^ 

Lpplledfrom the separating portion 601 and the pixels corresponding to the foreground supph^^^^^^ 

and outputs the synthesized foreground component image. Since the foreground area and the background area are 

nSt oveSapped. the synthesizer 603 applies, for example, logical sum operation to the foreground components and 

the pixels corresponding to the foreground to synthesize the foreground component image 

[0432] In initial process to be fiist perfomied in the process of synthesizing the foreground component .rnagejhe 

Ltheslzer 603 ^ores the Image of which all the pixels In the built-in frame memoir are 0 and ,n the Process o 

J^mhesizlng the foreg«>und component image, the synthesizer 603 stores {ovenvntes) ^'^^^^^'^^^^^^^^ 

image. Therefore, 0 Is stored as the pixel values In the pixels corresponding to the background area and In the fore- 

around component images supplied from the synthesizer 603. ^ , ^ ^^„r^/^ 

?oS The synthesizer 605 synthesizes the background component image on the basis of the background compo- 
neMplied from the separating portion 601 and the pixels corresponding to the background supplied from the 
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C15=B15/v+F09/v+F08/v+ F07/v 
=al 5 • Bl 5+ F09/V+F08 /V+F07/V 
* =al5»P15+F09/v+F08/v+F07/v (68) 

[0447] Here, a1 5 Is a mixture ratio of the fifteenth pixel from the leftmost in the frame #n. P1 5 is a pixel value of the 
10 fifteenth pixel from the leftmost in the frame #n-1 . 

[0448] On the basis of equation (68). sum f15 of the foreground components of the fifteenth pixel from the leftmost 
in the frame #n is expressed by equation (69). 

^5 fl5«F09/v+F08/v+F07/v 

=C15-al5-P15 (69) 

20 [0449] Similarly, sum f16 of the foreground components of the sixteenth pixel from the leftmost in the frame #n Is 
expressed by equation (70), and sum f 1 7 of the foreground components of the seventeenth pixel from the leftmost in 
the frame #n is expressed by equation (71 ). 

f16=C16-o16.P16 (70) 

25 

f17=C17-a17.P17 (71) 

30 [0450] Like the above, the foreground component fc included in the pixel value C of pixels belonging to the covered 
background area Is cateulated by equation (72). 

fc=C-a . P (72) 

3S 

[0451] P is the pixel value of the corresponding pixel in the frame one ahead. 

[0452] FIG, 93 is a diagram illustrating a process of separating the foreground components from the pixels belonging 
to the uncovered background area. In FIG. 93. a1 to a1 8 are the mixture ratios corresponding to the respective pixels 
in the frame #n. In FIG. 93, the second to fourth pixels from the leftmost belong to the uncovered background area. 
40 [0453] The pixel value C02 of the second pixel from the leftmost In the frame #n is expressed by equation (73). 

C02=B02/v+B02/v+B02/v+F01/v 
^a2-B02+F01/v 

45 

=a2-N02+F01/v (73) 

[0454] Here, a2 is the mixture ratio of the second pixel from the leftmost In the frame #n. N02 Is the pixel value of 
50 the second pixel from the leftmost in the frame #n+1 . 

[0455] On the basis of equation (73), sum f02 of the foreground components of the second pixel from the leftmost 
in the frame #n is expressed by equation (74). 

55 f02=F01/v 

=C02-a2-N02 (74) 
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components is 8, the number of the pixels conresponding to the unit of processing A Is 1 2. the amount of movement v 
is 5 and the virtual division number is 5, subsequent equations generated by the equation generator 803 are explained. 
[0498] When the foreground component included in the foreground component Image corresponding to the shutter 
time/v lies between F01/v and F08/V, the relations between FOI/v to F08/v and the pixel values C01 to C12 are ex* 
pressed in equations (78) to (89). 

C01=F01/v (78) 
C02=F02/V+F01/V (79) 
C03=F03/v+F02/v+F01/v (80) 
C04=F04/v+F03/v+F02/v+F01 N (81 ) 

C05=F05Av+F04/v+F03/v+F02^+F01/V (82) 
C06=F08Av+F05/v+F04Af+F03A/+F02/v (83) 
C07=F07/v+F06/v+F05/v+F04/v+F03^ (84) 
C08=F08Av+F07/V+F06/V+F05Av+F04yV (85) 
C09=F08/V+F07/V+F06A^+F05^ (86) 
C10=F08/V+F07/V+F06/v (87) 
C11=F08/V+F07/V (88) 
C12=F0aAr (89) 



[0499] The equation generator 803 generates equations by modifying the generated equations. Equations generated 
by the equation generator 803 are expressed in equations (90) to (101). 

45 

C01=1 . F01/V+0 . F02/V+0 • F03/V+0 • F04^4.0 • F05/V 

+0 . F06A(+0 • F07/V+0 • FOB/v (90) 

50 

C02=1 . F01/V+1 . F02/V+0 . F03/V+0 • F04/V+0 • F05/V 

+0 . F06/V+0 . F07/V+0 • F08/V (91) 

55 

C03=1 . F01/V+1 . F02^+1 . F03/V+0 • F04/V+0 • F06/v 
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+0 . F06/V+0 . F07/V+0 • FOB/v (92) 

C04=1 . F01/V+1 . F02/V+1 • FOa/V+l • F04/V+0 • F05/V 

+0 . F06/V+0 . F07/V+0 - FOB/v (93) 

C05=1 . F01/V+1 . F02^+1 . F03/V+1 • F04/V+1 • F05/V 

+0 . F06/V+0 . F07/V+0 • F08/V (94) 

C06=0 . F01/V+1 . F02/V+1 • F03/V+1 • F04/V+1 • F05/V 

+1 . F06/V+0 • F07/V+0 • FOB/v (95) 

C07=0 . F01/V+0 . F02/V+1 • FOa/V+l • F04/V+1 • F05/V 

+1 . F06/V+1 . F07/V+0 . FOB/v (96) 

C08=0 ■ F01/V+0 . F02/V+0 • F03A^+1 - F04A<+1 - F05^ 

+1 . F06A/+1 . F07/V+1 . FOB/v (97) 

C09=0 . F01/V+0 . F02/V+0 • F03/V+0 • F04/V+1 • FOS/v 

+1 . F06/V+1 . F07/V+1 . FOB/V (9B) 

C10=0 . F01/V+0 . F02^+0 . F03/V+0 • F04A^+0 • F05A^ 

+1 . F06A(+1 . F07/V+1 . FOBA^ (99) 

C1 1 =0 . F01/V+0 • F02/V+0 • F03A/+0 • F04/V+0 • F06/V 

+0 . F06A^+1 • F07/V+1 • FOB/v (100) 

CI 2=0 • F01/V+0 . F02A/+0 • F03/V+0 • F04/V+0 • F05/V 

+0 . F06/V+0 . F07/V+1 ■ FOB/v (101) 

[0500] Equations (90) to (1 02) may be expressed as an equation (1 02). 



08 

Cj^^aiJ^FUv (102) 



[0501] In equation (1 02), j means thie position of pixel. In this example, j is one of 1 to 1 2. Further, i means the position 
of foreground value. In this example, I is one of 1 to 8. alj is 0 or 1 according to the values of i and j. 
[0502] Considering an error, equation (102) is expressed as equation (103). 
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as 

Cj=2aiJ'Fih + ej (203) 

[0503] In equation (103). ej is an error included in indicated pixel Cj. 
[0504] Equation 103 can be expressed as equation 104 again. 

OS 

ej^Cj^Ysaij^Filv (104) 

[0505] Here, sum of the square of en-ors E is defined as shown in equation (105) to apply the least square method. 



E^^ej' (105) 

[0S06] In order to minimize the error, the value of the partial differential by the variable Fk to the sum of the square 
of en-ors E should equal zero. Fk satisfying equation (106) is obtained. 

dE , ^ . dej 

n cs 

= 2 'l,{(Cj 't.aij'FilvH-akjlv) = 0 (106) 

[0507] In equation (1 06). because the amount of movement v is fixed, equation (1 07) can be derived. 

f akj ' (Cj -Zaij -Fih) = 0 (107) 

JsOJ is 01 

[0506] Equation (1 06) is obtained by expanding and transposing equation (1 07). 

22 OS i2 

2 (akj ^Zaij-Fi) = v-D akj • Cj (los) 

JsOJ i^OI J^OJ 

[0509] Equation (1 08) is expanded In 8 equations obtained by substituting one of integers between 1 and 8 for k in 
equation (1 08). The obtained 8 equations can be expressed as a matrix equation, and this matrix equation is called a 
normal equation. 

[0510] An example of a nomial equation generated by the equation generator 803 based on the least square method 
is shown in equation (109). 
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[0511] If equation (109) Is expressed as A-F=v-C, C, A and v are known and F Is unknown. A and v are known at 
the time of model fomiing and C is already known by Inputting the pixel values In adding operation. 
[0512] Errors included in pixel C can be dispersed by calculating the foreground component by using the nonnai 
equation based on the least square method. 

[0513] The equation generator 803 supplies the nomnal equation generated by the above to the adder 804. 
[051 4] The adder 804 sets the pixel values C included In foreground component image in the matrix equation supplied 
from the equation generator 803 on the basis of the unit of processing supplied from the unit of processing detemnlnlng 
portion 801 . The adder 804 supplies the matrix where the pixel values C are set to the calculator 805. 
[0515] The calculator 805 calculates the foreground component Fi/v from which motion blur is eliminated through 
the processing based on the sweeping-out methods such as Gauss-Jordan elimination method, calculates Fi corre- 
sponding to the pixel value i of the foreground from which motion blur Is eliminated, wherein i is one of integers between 
0 and 8, and then outputs the foreground component Image from which motion blur Is eliminated, along with the cal- 
culated pixel value Fi to the motion blur adder 806 and the selector 807. 

[0516] Further, as shown in FIG. 100, for making the position of the foreground component image unchanged with 
respect to the screen, each of F01 to F08 is set to each of C03 to C1 0 in the foreground component image from which 
motion blur is eliminated. However, it may con-espond to arbitrary positions. 

[0517] The motion blur adder 806 may adjust the amount of motion blur by adding the amount by which motion blur 
is adjusted v' which is different from the amount of movement v, for example, the amount by which motion blur is 
adjusted v* Is half of the amount of movement v, or has no relation with the amount of movement v. For example, as 
shown in FIG. 101 , the motion blur adder 806 calculates the foreground component Fi/v' by dividing the foreground 
pixel value from which motion blur is eliminated Fi with the amount by which motion blur is adjusted v', calculates the 
sum of the foreground components FiA^, and then generates the pixel value where the amount of motion blur is adjusted. 
For example, when the amount by which motion blur is adjusted is 3, the pixel value C02 becomes (F01)/v', the pixel 
value COS becomes (F01+F02)/v', the pixel value C04 becomes (F01+F02+F03)/v' and the pixel value COS becomes 
(F02+F03+F04)//. 

[0518] The motion blur adder 806 supplies the foreground component image of which the amount of motion blur is 
adjusted to the selector 807. 

[0519] The selector 807 selects one of the foreground component image from which motion blur is eliminated and 
which is supplied from the calculator 805 and the foreground component image of which the amount of motion blur is 
adjusted and which Is supplied from the motion blur adder 806 on the basis of the selection signal selected by an user, 
and then outputs the selected foreground component image. 

[0520] Like the above, the motion blur adjusting unit 106 is able to adjust the amount of motion blur based on the 
selection signal and the amount v' by which motion blur is adjusted. 

[0521 ] Further, for example, as shown In FIG. 1 02, when the number of pixels corresponding to the unit of processing 
is 8 and the amount of movement v Is 4, the motion blur adjusting unit 106 generates the matrix equation shown in 

equation (110). 
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Includes the motion blur based on the motion vector and the positional information thereof. 
[0533] That is. it is possible to adjust the amount of motion blur included in the pixel values which are sample data. 
[0534] FIG. 105 is a block diagram showing another example of a configuration of themotion blur adjusting unit 106. 
The motion vector and the positional Information thereof supplied from the motion detector 102 are supplied to a unit 
5 of processing detemfilning portion 901 and a adjusting portion 905, and the area infomnatlon supplied from the area 
specifying unit 103 is supplied to the unit of processing detennining portion 901. The foreground component image 
supplied from the foreground/background separator 105 Is supplied to calculator 904. 

[0535] The unit of processing detennining portion 901 generates a unit of processing based on the motion vector 
and positional infonmation thereof, and the area information, and then supplies the generated unit of processing to the 

10 model fomriing portion 902 along with the motion vector. 

[0536] The model fonning portion 902 performs the model fomning based on the motion vector and the input unit of 
processing. More specifically, the model forming portion 902 stores, for example, a plurality of models corresponding 
to the number of pixels included in the unit of processing, the virtual division number of the pixel value in time direction 
and the number of the foreground components for each pixel in advance, selects a specifying model corresponding to 

15 the pixel value and the foreground component shown in FIG. 1 06 on the basis of the unit of processing and the virtual 
division number of the pixel value in time direction. 

[0537] For example, if the number of pixels corresponding to the unit of processing is 1 2 and the amount of movement 
V Is 5, the model forming portion 902 selects the model, totally comprised 8 foreground components and where the 
number of division is 5, the leftmost pixel includes one foreground component, the second pixel from the leftmost 

20 includes two foreground components, the third pixel from the leftmost includes three foreground components, the fourth 
pixel from the leftmost includes four foreground components, the fifth pixel from the leftmost includes five foreground 
components, the sixth pixel from the leftmost includes five foreground components, the seventh pixel from the leftmost 
Includes five foreground components, the eighth pixel from the leftmost Includes five foreground components, the ninth 
pixel from the leftmost includes fourforeground components, the tenth pixel from the leftmost Includes three foreground 

25 components, the eleventh pixel from the leftmost includes two foreground components, the twelfth pixel from the left- 
most includes one foreground component. 

[0538] Further, the model forming portion 902 may not select one of the models stored in advance, but generates a 
model based on the motion vector and the unit of processing when the motion vector and the unit of processing are 
supplied. 

30 [0539] The equation generator 903 generates equations based on the model supplied from the model fomning portion 
902. 

[0540] Referring to models of the foreground component image shown in FIGs, 1 06 to 1 08, examples of the equations 
generated by the equation generator 903 Is explained when the number of the foreground components is 8, the number 
of the pixels con-esponding to the unit of processing is 12 and the amount of movement v is 5. 
35 [0541] When the foreground component con-esponding to (shutter time)/v included in the foreground component 
image is between FOI/v and F08/v. the relationship between FOI/v to F08/v and the pixel values C01 to C12 are 
expressed In equations (78) to (89) as mentioned eartier. 

[0542] Paying attention to the pixel values CI 2 and 01 1 , the pixel value 01 2 comprise only the foreground component 
F08/V as shown In equation (111) and the pixel value 011 comprise the sum of the foreground components F08/v and 
40 F07/V. Therefore, the foreground component F07/v can be obtained from equation (112). 

F08^=C12 (111) 



F07/v=O11-C12 (112) 

[0543] In the same manner, referring to the foreground components included in the pixel values CIO to 001, the 
foreground components FOe/v to FOI/v can be obtained by the equations (113) to (118). 

50 

F06/v=C10-C11 (113) 



F05/v=O09-C10 (114) 
F04/v=C08-O09 (115) 
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responding to the unit of processing are not extracted, the procedure returns to step S904 and the processing for 
extracting the foreground components is repeated. 

[0557] In step S905, if it is determined that ali the foreground components corresponding to unit of processing are 
extracted, the procedure proceeds to step S906, and the adjusting portion 905 conwts each of the foreground com- 
ponents FOI/v to F08/V supplied from the calculator 904 based on the amount v of movement and then calculates the 
pixel values F01 to F08 from which the motion biur is eliminated. 

[0558] In step S907. the motion blur adder 906 calculates the pixel values of the foreground where amount of motion 
blur is adjusted, the selector 907 selects either the Image in which amount of movement is eliminated or the Image in 
which amount of movement is adjusted and outputs the selected image, and then the procedure is finished. 
[0559] Lilce the above, the motion blur adjusting unit 106 having the configuration shown in FIG. 105 can adjust 
motion blur by foreground image In which motion blur Is included more rapidly by more simple operations. 
[0560] Conventional methods for eliminating motion blur such as Wiener-filter, etc., are effective In ideal conditions, 
but are not effective for the quantized real images having noises. However, this method using the motion blur adjusting 
unit 106 having the structure shown in FIG. 105 is sufficiently effective for the quantized real images having noises 
and it is possible to eliminate the motion blur precisely. 

[0561] As mentioned above, the separation sender 1 1 having the stnjcture shown In FIG. 27 can adjust the amount 
of motion blur belonging to the input image. 

[0562] FIG. 1 1 0 is a block diagram showing another configuration of the separation server 11 . 

[0563] The same portions as those in FIG. 27 are gwen with the same reference numerals and the description thereof 

will be omitted adequately. 

[0564] The area specifying unit 1 03 supplies the area infonnation to the mixture ratio calculator 1 04 and the synthe- 
sizer 1001. 

[0565] The mixture ratio calculator 1 04 supplies the mixture ratio o to the foreground/background separator 105 and 
the synthesizer 1001. 

[0566] The foreground/background separator 1 05 the foreground component image to the synthesizer 1 001 . 
[0567] The synthesizer 1001 synthesizes any background image and the foreground component image supplied 
from the foreground/background separator 1 05 based on the mixture ratio a supplied from the mixture ratio calculator 
1 04 and the area information supplied from the area specifying unit 1 03, and then output the synthesized image cor- 
responding to the background image or the foreground component image. 

[0568] FIG. 1 1 1 Is a drawing showing a configuration of the synthesizer 1 001 . The background component generator 
1 021 generates the background component image based on the mixture ratio a and any background image, and then 
supplies the generated background component image to the mixed area image synthesizing portion 1 022. 
[0569] The mixed area image synthesizing portion 1 022 generates the mixed area synthesized image by synthesizing 
the background component Image supplied from the background component generator 1 021 and the foreground com- 
ponent and then supplies the generated mixed area synthesized Image to the image synthesizing portion 1023. 
[0570] The image synthesizing portion 1 023 synthesizes the foreground component Image, the mixed area synthe- 
sized image supplied from the mixed area image synthesizing portion 1022 and any background image based on the 
area Information to generate and output the synthesized image. 

[0571] Like the above, the synthesizer 1 001 can synthesize the foreground component image and any background 
Image. 

[0572] The image obtained by synthesizing the foreground component image and any background image based on 
the feature amount of the mixture ratio a is more natural when compared to the image obtained by synthesizing only 
the pixels. 

[0573] FIG. 112 is a block diagram showing another configuration of the separation server 11 which has adjusted 
the amount of motion blur. The separation sender 1 1 shown in FIG. 27 performs the area specification and calculates 
of the mixture ratio a serially, while the separation server 11 shown in FIG. 112 performs the area specification and 
calculates of the mixture ratio a in parallel. 

[0574] The same portions as those In FIG. 27 are given with the same reference numerals and the description thereof 
will be omitted. 

[0575] The input Image is supplied to the mixture ratio calculator 11 01 , the foreground/background separator 1102, 
the area specifying unit 103 and the object extracting unit 1 01 . 

[0576] The mixture ratio calculator 1101 calculates an estimated mixture ratio supposing that the pixel belongs to 
the covered background area and calculates an estimated mixture ratio supposing that the pixel belongs to the uncov- 
ered background area for each pixel belonging to the Input image on the basis of the input Image, and supplies the 
estimated mixture ratio for the case supposing that the pixel belongs to the covered background area and the estimated 
mixture ratio in the case that supposes the pixel belongs to the uncovered background area to the foreground/back- 
ground separator 1 1 02. 

[0577] FIG. 1 1 3 is a block diagram showing an example of a configuration of the estimated mixture ratio calculator 
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1101. 

[0578] An estimated mixture ratio processor 401 shown in FIG, 11 3 is the same as the estimated mixture ratio proc- 
essor 401 shown in FIG. 72. An estimated mixture ratio processor 402 shown in FIG. 113 is the same as the estimated 
mixture ratio processor 402 shown In FIG. 72. 
5 [0579] The estimated mixture ratio processor 401 calculates an estimated mixture ratio for each pixel using the 
calculation con-esponding to a model of the covered background area based on the Input image, and then outputs the 
calculated estimated mixture ratio. 

[0580] The estimated mixture ratio processor 402 calculates an estimated mixture ratio for each pixel using the 
calculation con'esponding to a model of the uncovered background area based on the input image, and then outputs 

10 the calculated estimated mixture ratio. 

[0581] The foreground/background separator 11 02 generates a foreground component Image from the Input image, 
based on the estimated mixture ratio for the case that supposes the pixel belongs to the covered background area, 
the estimated mixture ratio for the case that supposes the pixel belongs to the uncovered background area whteh are 
supplied from the mixture ratio calculator 1101, and based on the area Infonnation supplied from the area specifying 

IS unit 1 03, and then supplies the generated foreground component image to the motion blur adjusting unit 1 06 and the 
selector 1 07. 

[0582] FIG. 1 1 4 is a block diagram showing an example of a configuration the foreground/background separator 1 1 02 . 
[0583] The same portions as those in the foreground/background separator 1 05 shown In FIG. 89 are given with the 
same reference numerals and the description thereof will be omitted. 
20 [0584] The selector 1121 selects the estimated mixture ratio for the case that supposes the pixel belongs to the 
covered background area or the estimated mixture ratio for the case that supposes the pixel belongs to the uncovered 
background area which are supplied from the mixture ratio calculator 1101, based on the area information supplied 
from the area specifying unit 1 03, and then supplies the selected estimated mixture ratio as the mixture ratio a to the 
separating portion 601 . 

25 [0585] The separating portion 601 extracts the foreground component and background component from the pixel 
values of the pixel belonging to the mixed area based on the mixture ratio and the area Information supplied from the 
selector 1121 , supplies the extracted foreground component to the synthesizer 603 and supplies the extracted back- 
ground component to the synthesizer 605. 

[0586] The separating portion 601 can have the same configuration shown in FIG. 94. 
30 [0587] The synthesizer 603 synthesizes and outputs the foreground component Image. The synthesizer 605 syn- 
thesizes and outputs the background component Image. 

[0588] The motion blur adjusting unit 1 06 shown In FIG. 112, which may have the same structure as the configuration 
shown in FIG. 27, adjusts the amount of motion blur belonging to the foreground component image supplied from the 
foreground/background separator 1102 based on the area Infonnation and the motion vector, and then outputs the 

35 foreground component Image where the amount of motion blur Is adjusted. 

[0589] The selector 1 07 shown in FIG. 1 1 2 selects either the foreground component image supplied from the fore- 
ground/background separator 1 1 02 or the foreground component image in which the motion blur is adjusted supplied 
from the motion blur adjusting unit 106 based on the selection signal corresponding the selection of a user, and then 
output the selected foreground component Image. 

40 [0590] Like the above, the separation server 11 having the configuration shown In FIG. 112 can adjust the amount 
of motion blur Included in the image con-esponding to the foreground object included In the input image and outputs 
it. The separation server 11 having the configuration shown In FIG. 112 can calculate a mixture ratio a of burled Infor- 
mation and output the calculated mixture ratio a as the same manner In the first embodiment. 
[0591] FIG. 115 is a block diagram showing another configuration of the separation server 11 for synthesizing the 

45 foreground component Image and any background image. The separation server 11 shown in FIG. 110 performs the 
area specif teatlon and the cateulatlon of the mixture ratio a serially. But the separation server 11 shown In FIG. 115 
perfonns the area specification and the calculation of the mixture ratio a In parallel. 

[0592] The same portions as those in FIG. 112 are given with the same reference numerals and the description 
thereof will be omitted, 

50 [0593] The mixture ratio calculator 1101 shown in FIG. 115 calculates an estimated mixture ratio for the case that 
supposes the pixel belongs to the covered background area and an estimated mixture ratio for the case that supposes 
the pixel belongs to the uncovered background area for each pixel belonging to the Input image on the basis of the 
input image, and supplies the estimated mixture ratio for the case that supposes the pixel belongs to the covered 
background area and the estimated mixture ratio for the case that supposes the pixel belongs to the uncovered back- 

55 ground area to the foreground/background separator 11 02 and the synthesizer 1201 . 

[0594] The foreground/background separator 1102 shown in FIG. 115 generates a foreground component image 
from the input image based on the estimated mixture ratio for the case that supposes the pixel belongs to the covered 
background area and the estimated mixture ratio for the case that supposes the pixel belongs to the uncovered back- 
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receives the foreground component Image, the background component image and ID thereof transmitted from the 
separation server 11, stores them in its own storage unit 48 (see FIG. 2) and prints out them as needed. Further, the 
client computer 27 may make the separation server 11 store in Its own storage unit the foreground component image 
and the background component image separated by the separation server 11 or output them to the storage server 18 

5 via the network 1 so as to store them according to the instructions of a user. 

[0625] In the above explanation, the separating process has been described when the servtee fee has been paid to 
the financial transaction servers 25, 26 by the account charge server 24. However, the charge processing may be 
perfomned, by storing points, such as pr&^ald points, lndk:atlng that the user has previously paid the service fee to 
the provider of the separating service in the storage unit 48 (see FIG. 2) and deducting the points at every time the 

10 user is provided with the separating service. 

[0626] Here, refening to a flowchart of FIG. 120, the charge processing using the pre-paid points will be explained. 
[0827] In step SI 201. the client computer 27 specifies the service and transmits the ID infomiation and the verification 
infonnation. That Is, the client computer 27 perfomis the same processing as that in step S1101 of FIG. 117. 
[0628] In step S1211 , the charge processor 11a of the separation server 11 receives the ID infomnation and the 

IS verification Information. In step S1212, the charge processor 11a subtracts the points corresponding to the service fee 
for the separation processing from the pre-paid points which is stored In storage unit 48 con^spondlng to the amount 
of money previously paid by the user of the client computer 27 and stores it. In step SI 21 3, the separation server 11 
provides the service. That is, at this time, the separation server 11 perfomis the separation processing of the input 
image and transmits the foreground component Image and the background component image separated to the client 

20 computer 27. 

[0629] In step SI 202, the client computer 27 receives the provided service. That is, at this time, the client computer 
27, receives the foreground component Image and the background component Image transmitted from the separation 
server 11. 

[0630] Further, In the above explanation, a case that the separation server 11 stores the pre-paid points In its own 
25 storage unit 48 (see FtG. 2) has been described but for example, when a card in whteh the pre-paid points are stored, 
so called a pre-paid card is used, the same processing is perfonned. In this case, it is required that the client computer 
27 read out and transmits the pre-paid points stored in the pre-paid card in step SI 201, the separation server 11 
subtract the points corresponding to the service fee from the points received by the charge processing and the points 
subtracted is transmitted to the client computer to update the pre-paid card. 
30 [0631] Next, referring to the flowchart of FIG. 121 , the processing of the motion detecting servtoe for obtaining the 
motion vector and the positional infonnation of the Image specified by the client computer 27 will be explained. 
[0632] In step SI 301 , the client computer 27 outputs the Infomiation specifying an image to the motion detecting 
server 12. That is, as the infonnation specifying the Image desired by a user to be processed by the motion detecting 
process, the specific image or the Image ID specifying the image is output to the motion detecting sen/er 12. 
35 [0633] In step SI 31 1 , the motion detecting server 1 2 obtains the specified image. That is, the motion detecting server 

12 reads out and obtains the very image when an image is transmitted from the client computer 27 or the image 
con^esponding to the image ID when the specifk:atlon information is transmitted, via the networic 1 . 

[0634] In steps SI 31 2 and SI 321, the charge processor 12c of the motion detecting server 12 and the account 
charge server 24 perfomri the charge processing. Further, the charge processing Is similar to that in the separating 
40 service in FIG. 118 and FIG. 120 and thus, explanation thereof will be omitted. 

[0635] In step SI 31 3, the object extracting unit 12a of the motion detecting server 12 extracts each object from the 
specified Image obtained, and the motion detector 12b detects the positional infomiation and the motion vector, to 
transmit It to the client computer 27. 

[0636] In step SI 302. the client computer 27 receives and stores the positional information and the motion vector of 
45 the object transmitted from the motion detecting server 1 2. 

[0637] Further, the client computer 27 may store the positional Infomiation and the motion vector detected by the 
motion detecting server 12 in its own storage unit or output them to the storage server 18 via the network 1 so as to 
store them, in accordance with instructions of the user. 

[0638] Next, referring to the flowchart of FIG. 1 22, the processing of the area specifying service specifying area from 
50 the infomiation specifying the image and object input from the client computer 27, which Is perfomrjed by the area 
specifying server 13, will be explained. 

[0639] In step SI 401 , the client computer 27 outputs the infomiation specifying the image and the object to the area 
specifying server 13. That is, the infonnation specifying the object is output to the area specifying server 1 3 along with 
the specific image or the image ID specifying the image as the infomiation specifying an image desired by a user to 

55 area specify. 

[0640] |n step S1411 , the area specifying server 13 obtains the specified image. That is, the area specifying server 

13 reads out and obtains the very image when the image is transmitted from the client computer 27 and the image 
con^esponding to the Image ID when the Image ID specifying an Image Is transmitted, via the network 1 . 
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[0641] In steps S1 41 2 and SI 421 , the charge processor 13a of the area specifying server 13 and the account charge 
server 24 performs the charge processing. Further, the charge processing is similar to that in the separating service 
in FIG. 118 and FIG. 120 and thus, explanation thereof will be omitted. 

[0642] In step SI 41 3, the area specifying server 13 perfomis the area specifying processing on the basis of the 
information specifying the object. Further, the area specifying processing is similar to the processing explained referring 
to the flowchart of FIG. 53 and thus, explanation thereof will be omitted. 

[0643] In step S 1 41 4, the area specifying server 1 3 transmits the area infonnation obtained through the processing 
in step S1413 to the client computer 27. 

[0644] In step SI 402. the client computer 27 receives and stores the area information transmitted from the area 
specifying server 13. 

[0645] Further, the client computer 27 may store the area Infomiation obtained by the area specifying server 13 in 
its own storage unit or output it to the storage server 1 8 via the networic 1 so as to store it> in accordance with instructions 

of the user. 

[0646] Next, referring to a flowchart of FIG. 1 23, the processing of the mixture ratio calculating service of calculating 
the mixture ratio from the infonnatlon specifying the image and the object and the area Information input from the client 
computer 27, which is perfonmed by the mixture ratio calculating server 14, will be explained. 
[0647] In step SI 501 , the client computer 27 outputs, the infomiation specifying the image and the object and the 
area infomiation to the mixture ratio calculating server 14. 

That is, as the infomiation specifying an image of which mixture ratio is desired by a user to specify, the specific image 
or the image ID specifying the image, the infonnation specifying the object and the area information are output to the 
mixture ratio calculating server 14. 

[0648] In step S1511 , the mixture ratio calculating server 14 obtains the specified image. That is, the mixture ratio 
calculating server 14 reads out and obtains the very Image when the Image is transmitted from the client computer 27 
and the image con-esponding to the image ID when the image ID specifying an image is transmitted, via the network 1 . 
[0649] In steps SI 51 2 and SI 521 , the charge processor 1 4a of the mixture ratio calculating server 1 4 and the account 
charge server 24 performs the charge processing. Further, the charge processing is similar to that in the separating 
service in FIG. 118 and FIG. 120 and thus, explanation thereof will be omitted. 

[0650] In step S151 3, the mixture ratio calculating server 1 4 performs the mixture ratio calculation processing on the 
basis of the information specifying the object and the area infonmation. Further, the mixture ratio calculation processing 
Is similar to the processing explained referring to the flowchart of FIG. 81 and thus, explanation thereof will be omitted. 
[0651 ] In step S 1 51 4, the mixture ratio calculating server 1 4 transmits the mixture ratio obtained through the process- 
ing In step SI 51 3 to the client computer 27. 

[0652] In step S 1 502, the client computer 27 receives and stores the mixture ratio transmitted from the mixture ratio 
calculating server 14. 

[0653] Further, the client computer 27 may store the mixture ratio obtained by the mixture ratio calculating server 1 4 
in its own storage unit or output it to the storage server 18 via the network 1 so as to store it, in accordance with 
instructions of the user. 

[0654] Next, refemng to the flowchart of FIG. 1 24, the processing of service for separating the foreground component 
image and the background component image from the information specifying the image and the object, the area infor- 
mation and the mixture ratio input from the client computer 27, which is performed by the foreground/background Image 
separating server 15, will be explained. 

[0655] In step S1 601 , the client computer 27 outputs, the infonmation specifying the Image and the object, the area 
information and the infonnation of mixture ratio to the foreground/background image separating server 15. That is, as 
the information specifying an image desired by a user to separate into foreground and background, the concrete image 
or the image ID specifying the image, the infonnation specifying the object, the area infonmation and the information 
of mixture ratio are output to the foreground/background image separating server 15. 

[0656] In step S1611, the foreground/background Image separating server 15 obtains the specified image. That is, 
the foreground/background image separating server 15 reads out and obtains the very image when the image is trans- 
mitted from the client computer 27 and the image corresponding to the image ID when the image ID specifying an 
image Is transmitted, via the network 1 . 

[0657] In steps SI 61 2 and SI 621 , the charge processor 1 5a of the foreground/background image separating server 
1 5 and the account charge server 24 perf onns the charge processing. Further, the charge processing is similar to that 
in the separating service in FIG. 118 and FIG. 120 and thus, explanation thereof will be omitted. 
[0658] In step S1 61 3, the foreground/background image separating server 1 5 performs the foreground/background 
separating processing on the basis of the information specifying the object, the area Information and the mixture ratio. 
Further, the foreground/background separating processing is similar to the processing explained refemng to the flow- 
chart of FIG. 96 and thus, explanation thereof will be omitted. 

[0659] In step S1 614, the foreground/background image separating server 15 generates IDs to the foreground com- 
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specifying the image and the information specifying the object are output to the encoding server 17. 
[0675] In step S1 81 1 , the encoding server 1 7 obtains the specified image. That is, the encoding server 1 7 reads out 
and obtains the very image when the image is transmitted from the client computer 27 and the image corresponding 
to the image ID when the image ID specifying an Image is transmitted, via the network 1 . 
5 [0676] In steps S1B12 and S1821 , the charge processor 17a of the encoding server 17 and the account charge 
server 24 perfomn the charge processing. Further, the charge processing is similar to that in the separating service in 
FIG. 118 and FIG. 120 and thus, explanation thereof will be omitted. 

[0677] In step SI 81 3, the separation portion 2002 of the encoding server 1 7 perfomis the Image separating process- 
ing. Further, the Image separating processing is similar to the processing explained referring to step SI 01 3 in the 
10 flowchart of FIG. 117 and thus, explanation thereof will be omitted. 

[0678] In step 31 81 4, the encoding server 17 outputs the foreground component image and the background com- 
ponent image obtained through the processing In step 81813 to the storage server 18 so as to store them. In step 
SI 813, the storage server 18 stores the foreground component image and the background component image trans- 
mitted. 

IS [0679] In step SI 81 5, the encoding server 1 7 adds the motion vector and the positional information to the positional 
information of foreground component image and the positional Infomnation of background component Image generated 
through the encoding processing and transmits them to the client computer 27. 

[0680] In step SI 802, the client computer 27 receives and stores the positional Information of foreground component 
Image, the positional information of background component image, the motion vector and the positional infomriation 

20 transmitted from the encoding server 1 7. 

[0681] Further, in a case of separating and encoding the Input Image, the encoder 2001 may add only data corre- 
sponding to the difference to the code (positional Infonmation of the Image) of the image already encoded to output it 
when image similar to a pre-encoded image is encoded. For example, in a case of synthesizing the image shown In 
FIG. 128, when the encoding Infonnation of a first Image Including encoding Infomnation of a foreground component 

25 image 1 , a foreground component image 2 and a mixture ratio 1 and the encoding Information of a second Image 
including encoding Information of a foreground component Image 1 , a foreground component Image 3 and a mixture 
ratio 2 are synthesized, since the foreground component Image 1 is also Included in any image infonnation, the fore- 
ground component image 1 of any of the images may be omitted when synthesizing and consequently, the compression 
rate can be improved as much as the Information of the foreground component image 1 to be omitted, compared with 

30 a case that both images are simply synthesized. 

[0682] Consequently, when the first image and the second image shown in FIG. 128 are stored, if the first image is 
first stored, only the encoding Information of the mixture ratio 2 and the foreground component image 3 that is a dif- 
ference may be stored for the second Image. For this reason, when the encoding Information of the same image is 
stored repeatedly, the compression rate improves as the number of the stored images Increases. 

35 [0683] Further, the mixture ratio, the motion vector and the positional Infonnation to be encoded by the encoding 
server 17 may be the infonnation specified by a user, as shown In FIG. 129. Further, as shown in FIG. 129, for the 
Image to be encoded, the foreground component Image and the background component image corresponding to the 
image ID specified by the user may be read out from the storage server 18 and encoded. In this case, the encoding 
server 1 7 may not be provided with the separation portion 2002. 

40 [0684] Further, although the Image ID has been employed as the Information specifying image In the present inven- 
tion, the image positional Information may be employed Instead. 

[0685] Next, refemng to a flowchart of FIG. 1 30, the processing of service for synthesizing the specified images A 
and 6 from the infomnation specifying the images A and B input from the client computer 27, the motion vector, the 
mixture ratio, the positional information and the amount by which motion blur Is adjusted, which is perfonned by the 

45 synthesizing server 1 9, will be explained. 

[0686] in step S1901, the client computer 27 outputs the infonnation specifying the images A and B, the motion 
vector, the mixture ratio, the positional Infomnation and the infonnation of amount by which motion blur is adjusted to 
the synthesizing server 19. That is, as the infomnation specifying the Images A and B desired by a user to synthesize, 
the specific image or the image A ID and B ID (may be the aforementioned positional information of encoded Images) 

50 specifying the images A and B, the motion vector, the mixture ratio, the positional infonnation and the Information of 
amount by which motion blur is adjusted are output to the synthesizing server 1 9. 

[0687] in step S1911 , the synthesizing server 16 obtains the specified Images. That is, the synthesizing server 19 
reads out and obtains the very image when the Images are transmitted from the client computer 27 and the images 
corresponding to the image IDs when the image IDs specifying images are transmitted, via the network 1 . 
^ [0688] In steps SI 91 2 and SI 921 , the charge processor 1 9a of the synthesizing server 1 9 and the account charge 
server 24 perform the charge processing. Further, the charge processing is similar to that in the separating service In 
FIG. 118 and FIG. 120 and thus, explanation thereof will be onnltted. Further, the charge processing may be omitted 
In a case that the user accepting the encoding service has been charged by using the encoding server 16. Inversely, 
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In place of the user accepting the encoding sen^lce, the user accepting the synthesizing service may be charged. 
[0689] In step S1913, the synthesizing server 19 perfonns the processing of synthesizing the Images A and B on 
the basis of the motion vector, the mixture ratio, the positional information and the information of amount by which 
motion blur is adjusted. 

5 [0690] In step S1914, the synthesizing server 19 generates ID for the synthesized image {A+B) obtained through 
the processing in step SI 91 3 and transmits It to the client computer 27. 

[0691 ] in step SI 902, the client computer 27 receives and stores the synthesized image (A4-B) transmitted from the 
synthesizing server 19. 

[0692] Further, the client computer 27 may store the synthesized image (A+B) transmitted from the synthesizing 
10 server 19 in its own storage unit or output it to the storage server 1 8 via the network 1 so as to store it, in accordance 
with instmctions of the user. 

[0693] As described above, although the synthesizing server 20 Is made to be able to synthesize a plurality of images 
it may generate the encrypted image by adding the motion blur of the synthesized image using the amount by which 
motion blur Is adjusted as a l<ey. FIG. 1 31 shows a configuration of the motion blur adder 2021 for encryption, provided 

'5 to make the synthesizing sender 20 generate the encrypted image. 

[0694] The Input Infonnation processor 2031 of the motion blur adder for encryption 2021 outputs the Input signal to 
be encrypted to the imaging portion 2032, outputs infomiation of an encrypting key to the motion btur adder 2033 and 
outputs the image selection infonnation for selecting image (background component image) desired to synthesize by 
using the signal to be encrypted as the foreground component image to the synthesizing server 20, respectively. 

20 [0695] When the signal to be encrypted input from the input information processor 2031 is not Image signal, the 
imaging portion 2032 converts the signal Into the image signal to output It to the motion blur adder 2033. That Is, since 
the encrypted signal is supposed to be Image signal, the Imaging portion 2032 Images the signal that is not image 
signal in order to correspond with such processing. 

[0696] The motion blur adder 2033 generates the amount by which motion blur is adjusted on the basis of the infor- 
25 mation such as speed or direction input from the Input Infonnation processor 2031 and adds the motion blur to the 
Image signal Input from the Imaging portion 2032 to output it to the synthesizing server 20. The synthesizing server 
20 obtains the background component image on the basis of the image selection infonnation input from the input 
Infonnation processor 2031 and synthesizes the Image input from the motion blur adder 2033 as the foreground com- 
ponent Image with the obtained background component Image to generate and display the synthesized iniage. At this 
30 time, the image selection Information specifying the background component Image may be the background component 
Image itself and may be the positional information of background component image or ID of the background component 
Image. 

[0697] Next, referring to FIG. 132, the motion blur eliminating portion for encryption 2041 for decrypting the synthe- 
sized image encrypted by the motion blur adder for encryption 2021 provided in the synthesizing server 20 to covert 

35 It into the original signal will be explained. Further, the motion blur adder for encryption 2021 and the motion blur 
eliminating portion for encryption 2041 shown In FIG. 1 31 and FIG. 1 32 may be thought, for example, to be a functional 
block diagram of software built-in to the client computer 27 and to be a block diagram of hardware. Further, the motion 
blur adder for encryption 2021 and the motion blur eliminating portion for encryption 2041 may be constructed as an 
exclusive server on the network 1 . 

40 [0698] The separation server 1 1 separates the encrypted synthesized image into the foreground component image 
and the background component image and outputs the foreground component Image to which motion blur is added to 
the input infonnation processor 2051 . 

[0699] When the encrypted foreground component image input from the separation server 11 and the infonnation 
on speed and direction as a key for decrypting the encrypted foreground component image are input, the input infor- 
ms mation processor 2051 output them to the motion blur eiimlnating portion 2052. When the Image is displayed In 2-dj- 
mensions of x direction and y direction, the speed and the direction of the key are set thereto, respectively. 
[0700] The motion blur eliminating portion 2062 generates the amount of motion blur on the basis of the infonnation 
of speed and direction input from the input infomiation processor 2051 , perfonns the motion blur adding processing 
inverse to the motion blur adding processing performed by the motion blur adder for encryption 2021 on the encrypted 
50 foreground component Image, and decrypts the encrypted foreground component Image to output it to the signal con- 
verter 2053. When the encrypted signal to be encrypted Is not Image signal, the signal converter 2053 coverts the 
Image signal input from the motion blur eliminating portion 2052 into the original signal to output it. 
[0701] That Is, the motion blur adder 2033 (see FIG. 131) and the motion blur eliminating portion 2052 perfonns 
substantially the processing similar to the motion blur adder 806 in FIG. 97 and perf omns the motion blur adding process- 
55 ing inverse to each other using the amount by which motion blur is adjusted as the encrypting key. However, they are 
different in that gain-up processing is further performed on the motion blur adding processing first perfomied in the 
motion blur adding processes in x direction or y direction to be described below. 

[0702] Here, principles In which the image signal is encrypted by adding the motion blur will be explained. For ex- 
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B6X2/5 + ux2 + px3 -i- kx4. the pixel value Ky' = B7x3/5 + ux3 + px4 and the pixel value Py* = B8x4/5 + ux4. 
[0722] By performing all these processes in all the y direction, the synthesized image of which the background com- 
ponent image is the encrypted foreground component Image shown in FIG. 150 is generated. That is, the Input image 
of 5x5 pixels is converted into the image of 9x9 pixels (the pixels A to Y. the pixels Ax to Dx. the pixels Fx to Ix. the 
5 pixels Kx to Nx. the pixels Fx to Sx, the pixels Ux to Xx, the pixels A/ to T/, the pixels Ax* to Dx*. the pixels Fx* to ix*. 
the pixels Kx' to Nx* and the pixels Px' to Sx'). 

[0723] Further, since is the motion blur adding processing in the motion blur eliminating portion for encryption 2041 , 
in the reverse of the processing by the motion blur adder for encryption 202 1 , explanation of the decrypting processing 
will be omitted. 

10 [0724] Further, in the processing of step S2006, when encryption in the x direction is performed, the gain-up process- 
ing is perfonned and then, encryption in the y direction is perfonned. Therefore, it is necessary that after decryption in 
the y direction is performed, the gain is lowered, and then the decryption in the x direction is performed. Further, the 
order of the encryptions In the x direction and the y direction may be changed, l-lowever, since the gain-up processing 
is perfomned in the direction where the encryption is fomierty perfonmed, it is necessary that the order of decryptions 

IS should con-espond to the order of encryptions. 

[0725] Next, refening to the flowchart of FIG. 151 , the encrypting sen^ice by the synthesizing server 1 9 comprising 
the motion blur adder for encryption 2021 shown in FIG. 131 will be explained. Further, this is a processing that the 
client computer 27-1 connected to the network 1 transmits the signal to be encrypted to the synthesizing server 19, 
which encrypts it and transmits It to the client computer 27-2. Further, the client computer 27 may be provided with a 

20 hardware having a function of the Image separation processing of the separation server 11 having the motion blur 
eliminating portion for encryption 2041 or Installed with software having such function. 

[0726] In step S2101, the client computer 27-1 transmits infonmation desired to encrypt (signal to be encrypted), 
Infomiation of speed and direction as the encryption key and image selection infonnation (infonmation selecting the 
background component Image) to the synthesizing server 1 9. 

25 [0727] in step S21 1 1 , the motion blur adder for encryption 2021 of the synthesizing server 1 9 encrypte the infonnation 
desired to encrypt (signal to be encrypted) on the basis of the encryption key Input from the client computer 27-1 and 
performs the encrypting processing of synthesizing the selected background component image. Further, since the 
encrypting processing was explained referring to the flowchart of FIG. 136, explanation thereof will be omitted. 
[0728] I n step S21 1 2 . the synthesizing server 1 9 transmits a synthesized image encrypted by adding the motion blur 

30 to the client computer 27-1 . 

[0729] In step S21 02, the client computer 27-1 displays the synthesized image received from the synthesizing server 
19 to determine whether the Image Is desired by the user or not and If it Is determined that the Image Is the desired 
Image, it is notified to the synthesizing server 1 9 that the image Is the desired Image In step S21 03. In step S21 1 3, the 
synthesizing server 19 determines whether the Image is the desired image or not and for example, in this case, since 

35 it Is notified to the synthesizing server 19 that the Image is the desired Image In step S21 03, the procedure proceeds 
t0 8tepS2114. 

[0730] In step S2114, the charge processor 19a of the synthesizing server 19 and the account charge server 24 
perform the charge processing. Further, the charge processing is similar to that of the separating servtee in FIGs. 118 
and 120 and thus, explanation thereof will be omitted. , 
40 [0731 ] In step S21 1 5, the synthesizing server 1 9 transmits the synthesized image encrypted to the client computer 
27-1 . In step S21 04, the client computer 27-1 receives the encrypted synthesized Image and transmits it to the client 
computer 27-2. 

[0732] In step S21 41 , the client computer 27-2 receives the encrypted synthesized image. In step S2142, the client 
computer 27-2 determines whether the key Is Input or not and repeats this processing until the encryption key is input. 
45 If the information of speed and direction as the encryption key is input in step S2142, the motion blur eliminating portion 
2041 perfonns the motion blur processing on the basis of the speed and the direction Input In step 21 43. In step S21 44, 

the Image In which the motion blur is eliminated is displayed. 

[0733] If It is determined that the Image was not the desired Image in step S2102, it Is notified to the synthesizing 
server 19 In step S2015 that the image Is not the desired Image and the procedure returns to step S2101. Further, 
50 since it Is determined by this processing that the image Is not the desired image, the processing of the synthesizing 
server 1 9 returns to step S21 1 1 . 

[0734] That is, by this processing, when the key of speed and direction specified by a user is correctly input to the 
client computer 27-1 , an Image that the encrypted image has been correctly decrypted Is displayed. Further, through 
the same system as the aforementioned encrypting service, the decrypting service can be provided. 
55 [0735] Next, referring to FIG. 1 52, the correcting server 20 will be explained. 

[0736] The separation processor 1 1 of the correcting server 20 separates the input image (which Is may be the image 
ID and when the image is specified by the image ID, the corresponding image retrieved from the networi< 1) into the 
foreground component image and the background component Image to output the foreground component image to 
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21 reads out and obtains the image corresponding to the image iO transmitted from the client computer 27 via the 
network 1 . 

[0752] In steps S231 2 and S2321 , the charge processor 21 a of the purchasing server 21 and the account charge 
server 24 perfonn the charge processing. Further, the charge processing is similar to that in the separating service in 
5 FIGs. 1 1 8 and 1 20 and thus, explanation thereof will be omitted. 

[0753] In step S2313, the purchasing server 21 transmits the image obtained through the processing in step S2311 
to the client computer 27. 

[0754] In step S2302, the client computer 27 receives and stores the image transmitted from the purchasing server 21 . 
[0755] Also, the client computer 27 may store the Image purchased by the purchasing server 21 In its own storage 

10 unit or output it to the storage server 18 so as to store it via the network 1 in accordance with instructions of a user. 
Further, the client computer 27 may transmit the image to another client computer 27, for example, to present it. Fur- 
thermore, another user may purchase the foreground component image, the background component image, the syn- 
thesized image, the corrected image or the like each of wliich is separated by the separation processing service, the 
synthesizing service or the correcting service. 

'5 [0756] Next, referring to a flowchart of FIG. 156, the processing of the image selling service for selling the image 
specified by the client computer 27, which Is performed by the selling server 22, will be explained. 
[0757] In step S2401, the client computer 27 outputs the infonnation specifying the Image desired to sell to the 
purchasing server 21 . That is, the image desired by user to sell is output to the selling server 22. 
[0758] In step S241 1 , the selling server 22 obtains the Image desired to sell. That is. the selling server 22 obtains 

20 the image transmitted from the client computer 27. 

[0759] In steps S2422, the selling server 22 sets a proper price on the image desired to sell. The price may be set» 
for example, by user in advance, or through an auction on the network 1 . Further, if an image is a person, the price 
may be set according to whether the person Is a well-known figure or not when the image is selected. 
[0760] In step S2413 and S2431 , the charge processor 22a of the selling server 22 and the account charge server 

25 24 perform the charge processing. 

[0761] Here, referring to the flowchart of FIG. 157, the processing of the charge will be explained. Further, the real 
processing of the charge is performed by the selling server 22 and the account charge server 24. But, the information 
needed for various processing is also output from the client computer 27 and therefore the processing of the client 
computer 27 will also be explained here. 

30 [0762] in step S2501 , an ID for discerning a user (who user who Is selling an image) is transmitted to the selling 
server 22 via the network 1 . 

[0763] In step S2511 , the selling server 22 transmits a cost and an ID for discerning the selling server 22 to the 
account charge server 24 on the basis of the ID information transmitted from the client computer 27. 
[0764] In step S2521, the account charge server 24 requests the financial transaction server 26 having provider 
35 account to pay the cost to the financial transaction server 25 having customer account on the basis of the transmitted 
ID for discerning the selling server 22. 

[0765] In step S2531 , the financial transaction server for provider 26 transfers the cost corresponding to the amount 
of sale from the provider account to the financial transaction server for customer 25 having account of the customer. 
[0766] IHere, the flowchart of FIG. 156 will be explained again. 
40 [0767] In step S2424, the selling server 22 notifies the end of sale to the client computer 27. In step S2402, the client 
computer 27 receives the notification of the end of sale. 

[0768] Also, the selling sen/er 22 may make the selling server 21 store the image sold by the user in its own storage 
unit or output the image to the storage server 18 so as to store it. Further, when the price is set through an auction 
system as mentioned above, the selling server 22 may make the image sold by the user to the client computer of a 

45 successful bidder 27. 

[0769] Next, refenlng to the flowchart of FIG. 158, the retrieving server 23 will be explained. 
[0770] The retrieving server 23 retrieves the image which has been picked up on the CEunera terminal unit 1 connected 
to the networtc 1 on the basis of the retrieving conditions Input from the client computer 27 and output the request 
image. The retrieving condition is time, season, weather, area, position, subject or the like. 

50 [0771] The control unit 21 61 of the retrieving sen/er 23 controls the whole operation of the retrieving server 23. In 
database 2162, position data which is obtained by GPS 76b provided with a camera terminal unit 28 corresponding to 
the camera ID of each camera terminal unit 28 which the retrieving server 23 recognizes, weather data, subject data 
or the like 2062b are stored as database. The contents of the database 21 62 are obtained and updated from the camera 
terminal unit 28 via the network 1 by controlling the communteation unit 2165 at a predetermined time interval through 

55 the control unit 2161. 

[0772] The storage unit 21 63 stores the image obtained from the camera tenninal unit 28 on the networtc 1 from the 

communication unit 2165 or stores infonnation needed for the processing of various images. 

[0773] The requested Information generator 21 64 arranges the retrieving conditions input from the client computer 
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received by the requested information receiving means, onty to the first communication apparatus after the charge 
processing has been finished by the first charging means and the second charging means. 

[0813] The significant infomiation may include the foreground component image and the baclcground component 
image, the encoded data transmitting means may transmit the requested infomiation received by the requested infor- 
mation receiving means and the encoded data as the foreground component Image and the background component 
image generated by the separating means when the predetermined image is separated into the foreground component 
image and the background component image by the separating means, only to the first communication apparatus after 
the charge processing has been finished by the first charging means and the second charging means. 
[0814] The requested infomiation input means may input an image data ID discerning a predetenmined image data 
in addition to the requested information of the user, and the encoded data transmitting means may transmit the encoded 
data as signifrcant information generated when the separating means separates the predetennined image into the 
foreground component image and the background component image on the basis of the requested information and 
the predetermined image data con-esponding to the image data ID only to the first communication apparatus after the 
charge processing has been finished by the first charging means and the second charging means. 
[0815] The second communication apparatus may further comprise an area Information generating means for gen- 
erating area infomnation discerning any of a foreground area having foreground object components constituting a fore- 
ground object of the predetemiined image data corresponding to the image data ID, a background area having back- 
ground object components constituting a background object of the predetermined image data corresponding to the 
image data ID and a mixed area in which the foreground area and the background area are mixed, wherein the signif- 
icant infomnation includes area infonmation, and wherein the encoded data transmitting means may transmit the en- 
coded data as area infomnation generated by area Information generating means when the separating means separates 
the predetemnined image Into the foreground component Image and the background component image on the basis 
of the requested infomnation and the predetermined image data corresponding to the image data ID, only to the first 
communication apparatus after the charge processing has been finished by the first charging means and the second 
charging means. 

[081 6] The second communication apparatus may further comprise a mixture ratio generating means for generating 
a mixture ratio indicating a ratio of the mixed area of the predetermined image data corresponding to the image data 
ID on the basis of area information, wherein the significant infomnation includes the mixture ratio, and the encoded 
data transmitting means may transmit the encoded data as mixture ratio generated by the mixture ratio generating 
means when the separating means separates the predetemnined image into the foreground component image and the 
background component image on the basis of the requested information and the predetermined image data corre- 
sponding to the image data ID only to the first communication apparatus after the charge processing has been finished 
by the first charging means and the second charging means. 

[0817] The significant infomnation may Include the foreground component image and the background component 
Image, the encoded data transmitting means may transmit the requested information and the encoded data as the 
foreground component image and the background component image generated by the separating means when the 
predetemnined image corresponding to the image data ID Is separated into the foreground component image and the 
background component image by the separating means, only to the first communication apparatus after the charge 
processing has been finished by the first charging means and the second charging means. 

[0818] A communication method of the 1 ^ system of the present invention is characterized in that a communication 
method of the first communication apparatus comprises: a requested information Input step of Inputting requested 
infomnation of a user frohn among a plurality of encoded data which are generated on the basis of predetermined image 
data that Is obtained by an image pickup element having a predetemiined number of pixels, each having a time Inte- 
gration effect, and that is comprised of pixel values detemnined for every pixels in accordance with an amount of light 
forming an image integrated over time; a requested information transmitting step of transmitting the requested infor- 
mation which is input in the requested infonmation input step to the second communication apparatus; and an encoded 
data receiving step of receiving the encoded data which are generated on the basis of the Image data corresponding 
to the requested information transmitted from the second communication apparatus, and a communication method of 
the second communication apparatus comprises: a requested infomiation receiving step of receiving the requested 
information transmitted from the first communication apparatus; a separating step of separating the predetemnined 
image data into a foreground component Image having foreground object components constituting a foreground object 
and a background component image having background object components constituting a background object; and an 
encoded data transmitting step of transmitting to the first communication apparatus the encoded data which are gen- 
erated when the image is separated into the foreground component image and the background component image In 
the separating step on the basis of the requested infomiation received In the requested information receiving step. 
[0819] A program of a 2^ recording medium of the present invention is characterized in that a program controlling 
the first communication apparatus comprises: a requested infomnation input control step of controlling input of requested 
information of a user from among a plurality of encoded data which are generated on the basis of predetemiined image 
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data that is obtained by an image piclcup element having a predetemiined number of pixels, each having a time Inte- 
gration effect, and that is comprised of pixel values detemnined for every pixels In accordance with an amount of light 
forming an image integrated overtime; a requested infomiation transmission control step of controlling transmission 
of the requested infomfiation which is input In the requested infomiatlon input control step to the second communication 
apparatus; and an encoded data reception control step of controlling reception of the encoded data which are generated 
on the basis of the image data corresponding to the requested infonmation transmitted from the second communication 
apparatus, and a program controlling the second communication apparatus comprises: a requested infomiatlon re- 
ception control step of controlling reception of the requested information transmitted from the first communication ap- 
paratus; a separation control step of controlling separation of the predetermined image data into a foreground compo- 
nent Image having foreground object components constituting a foreground object and a baclcground component image 
having background object components constituting a background object; and an encoded data transmission control 
step of controlling transmission of the encoded data which are generated when the image is separated into the fore- 
ground component image and the background component image in the separation control step on the basis of the 
requested infomnation received in the requested information reception control step to the first communication apparatus. 
[0820] A 2"«^ program of the present invention is characterized by making a computer for controlling the first com- 
munteation apparatus perfonn: a requested information Input control step of controlling Input of requested Information 
of a user from among a plurality of encoded data which are generated on the basis of predetemiined image data that 
is obtained by an image pickup element having a predetemiined number of pixels, each having a time integration effect, 
and that is comprised of pixel values detemnined for every pixels In accordance with an amount of light forming an 
image Integrated over time; a requested infonmation transmission control step of controlling transmission of the re- 
quested infonmation which is input in the requested infomnation input control step to the second communication appa- 
ratus; and an encoded data reception control step of controlling reception of the encoded data which are generated • 
on the basis of the Image data corresponding to the requested Information transmitted from the second communfeation 
apparatus and by making a computer for controlling the second communication apparatus perfomi: a requested Infor- 
mation reception control step of controlling reception of the requested Infomiatlon transmitted from the first communi- 
cation apparatus; a separation control step of controlling separation of the predetemiined image data into a foreground 
component image having foreground object components constituting a foreground object and a background component 
image having background object components constituting a background object; and an encoded data transmission 
control step of controlling transmission of the encoded data which are generated when the Image is separated Into the 
foreground component image and the background component Image In the separation control step on the basis of the 
requested infonmation received In the requested information reception control step, to the first communication appa- 
ratus, 

[0821] A 2"** communication apparatus of the present Invention Is characterized by comprising: a requested infor- 
mation input means for inputting requested infonmation of a user from among a plurality of encoded data which are 
generated on the basis of predetermined Image data that is obtained by an image pickup element having a predeter- 
mined number of pixels, each having a time integration effect, and that is comprised of pixel values determined for 
every pixels in accordance with an amount of light fonming an image Integrated over time; a requested Information 
transmitting means for transmitting the requested information which Is input by the requested Infonmation input means 
to other communteatlon apparatus; and an encoded data receiving means for receiving the encoded data which are 
generated on the basis of the Image data corresponding to the requested infomnation transmitted from other commu- 
nication apparatus. 

[0822] A 2^^ communication method of the present invention is characterized by comprising: a requested information 
input step of inputting requested infomnation of a user from among a plurality of encoded data which are generated on 
the basis of predetemiined Image data that Is obtained by an image pickup element having a predetemiined number 
of pixels, each having a time Integration effect, and that is comprised of pixel values detemnined for every pixels In 
accordance with an amount of light forming an Image Integrated overtime; a requested Infonmation transmitting step 
of transmitting the requested information which is input in the requested infomnation input step to other communication 
apparatus; and an encoded data receiving step of receiving the encoded data which are generated on the basis of the 
image data corresponding to the requested infonmation transmitted from other communication apparatus. 
[0823] A program of a 3"^ recording medium of the present invention Is characterized by comprising: a requested 
Infonmation Input control step of controlling input of requested infonmation of a user from among a plurality of encoded 
data which are generated on the basis of predetemiined Image data that is obtained by an Image pickup element 
having a predetermined number of pixels, each having a time Integration effect, and that is comprised of pixel values 
detemnined for every pixels In accordance with an amount of light fomnlng an image Integrated over time; a requested 
Infonmation transmission control step of controlling transmission of the requested Infonmation which is input in the 
requested infonmation Input control step to other communication apparatus; and an encoded data reception control 
step of controlling reception of the encoded data which are generated on the basis of the Image data corresponding 
to the requested Infomnation transmitted from other communteatlon apparatus. 
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the basis of the charge information. 

[0843] The charging means may perform the charge processing by deducting the number of points corresponding 
to the cost information from the number of point of each user used in the charge processing and con'esponding to cash. 
[0844] The output means may output the synthesized image to the communication apparatus of the user having 
finished the charge processing after the charge processing has been finished by the charging means. 
[0845] A 4^ communication method of the present invention Is characterized by comprising: a requested Infomriatlon 
Input step of inputting the requested information of user; a synthesizing step of synthesizing a foreground component 
image having foreground object components constituting a foreground object and a baclcground component image 
having background object components constituting a baclcground object of predetenmined image data that is obtained 
by an image piclcup element having a predetemiined number of pixels, each having a time integration effect, and that 
is comprised of pixel values determined for every pixels In accordance with an amount of light forming an Image Inte- 
grated overtime in accordance with request Information and generating a synthesized image; and a synthesized image 
output step of outputting the synthesized image generated in the synthesizing step. 

[0846] A program of a 6^ recording medium of the present invention Is characterized by comprising: a requested 
infomiation input control step of controlling input of the requested information of user; a syntheslzation control step of 
controlling syntheslzation of a foreground component image having foreground object components constituting a fore- 
ground object and a background component Image having background object components constituting a background 
object of predetenTiined Image data that is obtained by an Image pickup element having a predetermined number of 
pixels, each having a time Integration effect, and that is comprised of pixel values determined for every pixels in ac- 
cordance with an amount of light fonning an image Integrated over time in accordance with request infomiation, to 
generate a synthesized image; and a synthesized Image output control step of controlling output of the synthesized 
image generated in the syntheslzation control step. 

[0847] A 6*^ program of the present invention is characterized by making a computer pert orni: a requested infonnation 
input control step of controlling input of the requested infonnation of user, a syntheslzation control step of controlling 
syntheslzation of aforeground component image having foreground object components constituting a foreground object 
and a background component Image having background object components constituting a background object of pre- 
determined Image data that Is obtained by an Image pickup element having a predetemnlned number of pixels, each 
having a time Integration effect, and that Is comprised of pixel values detemilned for every pixels in accordance with 
an amount of light fomiing an image integrated over time in accordance with request Infonnation, to generate a syn- 
thesized image; and a synthesized image output control step of controlling output of the synthesized Image generated 
in the syntheslzation control step. 

[0848] A 3^ communication system of the present invention is characterized in that the first communication apparatus 

comprises: a requested Information input means for Inputting request information of a user; a requested Infonnation 
transmitting means for transmitting the requested Infonnation which Is input by the requested infonnation Input means 
to the second communication apparatus; and a synthesized data receiving means for receiving the synthesized data 
transmitted from the second communication apparatus In accordance with the requested information, and the second 
communication apparatus comprises: a requested Infonnation receiving means for receiving the requested infonnation 
transmitted from tiie first communication apparatus; a synthesizing means for synthesizing a foreground component 
image having foreground object components constituting a foreground object and a background component image 
having background object components constituting a background object of predetemiined Image data that is obtained 
by an image pickup element having a predetermined number of pixels, each having a time Integration effect, and that 
is comprised of pixel values determined for every pixels In accordance with an amount of light forming an image Inte- 
grated overtime in accordance with the requested infonnation and generating tlie synthesized image; and a synthesized 
image transmitting means for transmitting the synthesized image generated by the synthesizing means to the first 
communication apparatus. 

[0849] The requested Infonnation input means may input a predetemiined foreground component image, a prede- 
termined background component image and the significant infonnation used when synthesizing the predetemiined 
foreground component image and the predetermined background component Image along with the requested infor- 
mation of the user, and the synthesizing means may synthesize the predetemiined foreground component Image and 
the predetermined background component image input by the requested information Input means along with the re- 
quested information on the basis of the significant information and generate the synthesized image. 
[0850] The requested infomiation input means may input a predetemiined foreground component image, a prede- 
tennined background component image and the mixture ratio of the mixed area where the predetemiined foreground 
component image and the predetermined background component image are mixed as significant information used 
when synthesizing the predetemiined foreground component Image and the predetemiined background component 
image along with the requested Information of the user, and the synthesizing means may synthesize the predetermined 
foreground component image and the predetemiined background component Image Input by the requested information 
input means along the requested Infonnation on the basis of the mixture ratio as significant information and generate 
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the synthesized Image. 

[0851] The requested information input means may input a predetermined foreground component image, a prede- 
temiined background component image, and amount of movement and moving direction of the foreground component 
image as significant infomiation used when synthesizing the predetemiined foreground component image and the 
predetermined bacl<ground component image along with the requested Infomiation of the user, and the synthesizing 
means may adjust of the predetermined foreground component image input by the requested Information Input means 
along with the requested Information on the basis of amount of movement and the moving direction of the foreground 
component image as significant Information, synthesize the predetemiined foreground component image In which the 
motion blur is adjusted and the predetemiined background component image and generate the synthesized Image. 
[0852] The requested infomiation Input means may Input a predetemiined foreground component Image, a prede- 
termined background component image and initial positional information, amount of movement and moving direction 
of the foreground component image as significant information used when synthesizing the predetemiined foreground 
component Image and the predetenmined background component Image along with the requested infomiation of the 
user, and the synthesizing means may adjust the motion blur of the predetermined foreground component Image Input 
by the requested Information Input means along with the requested infomiation on the basis of amount of movement 
and the moving direction as significant infomiation, cateulate the mixture ratio, synthesize the predetermined foreground 
component image where motion blur Is adjusted and the predetemiined background component image using the cal- 
culated mixture ratio on the basis of the initial positional Infonmation, amount of movement and the moving direction 
of the foreground component Image as significant information, and then generate the synthesized Image. 
[0853] The requested infomiation input means may input a foreground component Image ID discerning a predeter- 
mined foreground component Image, a background component Image ID disceming a predetermined background com- 
ponent image and significant infomiation along with the requested infomiation of the user, and the synthesizing means 
may synthesize the foreground component Image corresponding to the foreground component Image ID and the back- 
ground component image corresponding to the background component image ID on the basis of the significant Infor- 
mation In accordance with the foreground component image ID disceming the predetermined foreground component 
Image, the background component image ID discerning the predetermined background component image, and signif- 
icant information along with the requested Infomiation of the user, and generate the synthesized image. 
[0854] The charging means for perfomiing charge processing In accordance with the requested Information may be 
further comprised. 

[0855] The charging means may generates charge Infomiation Including user ID, the communication apparatus ID 
and charge Information including the cost information con^espondlng to the requested infomiation In accordance with 
the requested Information. 

[0856] The charging means may perform the charge processing with respect to a financial account of the user on 
the basis of the charge Infomiation. 

[0857] The charging means may perform the charge processing by deducting the number of points con-espondlng 
to the cost infomiation from the number of points of each user used In the charge processing. 
[0858] The output means may output the synthesized Image to the communication apparatus of the user having 
finished the charge processing after the charge processing has been finished by the charging means. 
[0859] A communication method of the S"* communication system of the present Invention Is characterized In that 
a communication method of the first communteatlon apparatus comprises: a requested information input step of Input- 
ting request infomiation of a user; a requested information transmitting step of transmitting the requested infomiation 
which is input in the requested Information Input step to the second communication apparatus; and a synthesized data 
receiving step of receiving the synthesized data transmitted from the second communication apparatus in accordance 
with the requested information, and a communication method of the second communication apparatus comprises: a 
requested information receiving step of receiving the requested information transmitted from the first communication 
apparatus; a synthesizing step of synthesizing a foreground component image having foreground object components 
constituting a foreground object and a background component image having background object components consti- 
tuting a background object of predetermined Image data that Is obtained by an Image pickup element having a prede- 
termined number of pixels, each having a time integration effect, and that is comprised of pixel values detemilned for 
every pixels in accordance with an amount of light fomning an image Integrated over time In accordance with the 
requested Information and generating the synthesized Image; and a synthesized image transmitting step of transmitting 
the synthesized image generated in the synthesizing step to the first communication apparatus. 
[0860] A program of a 7**^ recording medium of the present Invention is characterized In that a program controlling 
the first communication apparatus comprises: a requested information input control step of controlling input of request 
infomiation of a user; a requested information transmission control step of controlling transmission of the requested 
information which is input In the requested information Input control step to the second communication apparatus; and 
a synthesized data reception control step of controlling reception of the synthesized data transmitted from the second 
communication apparatus in accordance with the requested Infomiation, and a program controlling the second com- 
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munication apparatus comprises: a requested information reception control step of controlting reception of the request- 
ed infonnation transmitted from the first communication apparatus: a synthesization control step of controlling synthe- 
sization of a foreground component image having foreground object components constituting a foreground object and 
a background component image having bacl<ground object components constituting a background object of predeter- 
mined image data that is obtained by an image pickup element having a predetemilned number of pixels, each having 
a time Integration effect, and that Is comprised of pixel values determined for every pixels in accordance with an amount 
of light forming an image integrated over time In accordance v/ith the requested Infomnation and generating the syn- 
thesized image; and a synthesized image transmission control step of controlling transmission of the synthesized Image 
generated in the synthesizing step to the first communteatlon apparatus. 

[0861] A 7^ program of the present Invention is characterized by making a computer for controlling the first commu- 
nication apparatus perform: a requested Infonmation input control step of controlling Input of request infomnation of a 
user; a requested information transmission control step of controlling transmission of the requested Information which 
is input in the requested information input control step to the second communication apparatus; and a synthesized 
data reception control step of controlling reception of the synthesized data transmitted from the second communication 
apparatus In accordance with the requested information, and by making a computer for controlling the second com- 
munication apparatus perform: a requested information reception control step of controlling reception of the requested 
information transmitted from the first communication apparatus; a synthesization control step of controlling synthesi- 
zation of a foreground component image having foreground object components constituting a foreground object and 
a background component image having background object components constituting a background object of predeter- 
mined image data that is obtained by an image pickup element having a predetermined number of pixels, each having 
a time integration effect, and that Is comprised of pixel values determined for every pixels in accordance with an amount 
of light fomning an image integrated over time in accordance with the requested information and generating the syn- 
thesized image; and a synthesized image transmission control step of controlling transmission of the synthesized Image 
generated in the synthesization control step to the first communication apparatus. 

[0862] A 5**» communication apparatus of the present invention Is characterized by comprising: a requested Informa- 
tion Input means for Inputting request Infonmation of a user; a requested Information transmitting means for transmitting 
the requested information which is Input by the requested information input means to other communication apparatus; 
and a synthesized data receiving means for receiving the synthesized data transmitted from other communtoatlon 
apparatus in accordance with the requested infomiation. 

[0863] A 5th communication method of the present Invention Is characterized by comprising: a requested infonmation 
input step of inputting request information of a user; a requested Information transmitting step of transmitting the re- 
quested Information which is input in the requested Information input step to other communication apparatus; and a 
synthesized data receiving step of receiving the synthesized data transmitted from other communication apparatus in 
accordance with the requested infomnation. 

[0864] A program of an 8^^ recording medium of the present invention is characterized by comprising: a requested 
Infonnatlon input control step of controlling input of request infomiation of a user; a requested infonmation transmission 
control step of controlling transmission of the requested Infonmation which Is Input In the requested Information input 
control step to other communication apparatus; and a synthesized data reception control step of controlling reception 
of the synthesized data transmitted from other communication apparatus in accordance with the requested Infonmation. 
[0865] An 8^^ program of the present invention is characterized by making a computer perfonm: a requested infor- 
mation Input control step of controlling input of request Information of a user; a requested information transmission 
control step of controlling transmission of the requested information which is Input in the requested Infonmation Input 
control step to other communication apparatus; and a synthesized data reception control step of controlling reception 
of the synthesized data transmitted from other communication apparatus In accordance with the requested infonmation. 
[0866] A 6^^ communication apparatus of the present invention is characterized by comprising: a requested informa- 
tion receiving means for receiving the requested infonmation transmitted from other communication apparatus; a syn- 
thesizing means for synthesizing a foreground component image having foreground object components constituting a 
foreground object and a background component image having background object components constituting a back- 
ground object of predetenmined image data that is obtained by an image pickup element having a predetermined 
number of pixels, each having a time integration effect, and that is comprised of pixel values detenmlned for every 
pixels In accordance with an amount of light forming an image Integrated over time In accordance with the requested 
infonmation and generating the synthesized image; and a synthesized image transmitting means for transmitting the 
synthesized image generated by the synthesizing means to other communication apparatus. 
[0867] A 6^ communk^atton method of the present invention Is characterized by comprising: a requested Infonmation 
receiving step of receiving the requested information transmitted from other communication apparatus; a synthesizing 
step of synthesizing a foreground component Image having foreground object components constituting a foreground 
object and a background component image having background object components constituting a background object 
of predetenmined image data that is obtained by an image pickup element having a predetermined number of pixels, 
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each having a time integration effect, and that is comprised of pixel values detemiined for every pixels in accordance 
with an amount of light fonning an image integrated over time in accordance with the requested infomnation and gen- 
erating the synthesized image; and a synthesized image transmitting step of transmitting the synthesized image gen- 
erated in the synthesizing step to other communication apparatus. 

[0868] A program of a 9*^ recording medium of the present invention Is characterized by comprising: a requested 
infomiation reception control step of controlling reception of the requested infonmation transmitted from other commu- 
nication apparatus; a synthesization control step of controlling syntheslzation of a foreground component Image having 
foreground object components constituting a foreground object and a background component image having back- 
ground object components constituting a background object of predetemiined image data that is obtained by an image 
pickup element having a predetemnined number of pixels, each having a time integration effect, and that is comprised 
of pixel values determined for every pixels in accordance with an amount of light forming an Image Integrated over 
time in accordance with the requested information and generating the synthesized image; and a synthesized image 
transmission control step of controlling transmission of the synthesized image generated in the synthesization control 
step to other communication apparatus. 

[0869] A 9^^ program of the present Invention is characterized by making a computer perf onm: a requested infonmation 
reception control step of controlling reception of the requested Infomiation transmitted from other communication ap- 
paratus; a synthesization control step of controlling synthesization of a foreground component image having foreground 
object components constituting a foreground object and a background component image having background object 
components constituting a background object of predetemiined image data that is obtained by an Image pickup element 
having a predetermined number of pixels, each having a time Integration effect, and that is comprised of pixel values 
detemilned for every pixels in accordance with an amount of light fonming an Image integrated overtime in accordance 
with the requested infomiation and generating the synthesized image; and a synthesized image transmission control 
step of controlling transmission of the synthesized image generated In the synthesization control step to other com- 
munication apparatus. 

[0870] A 7*^ communication apparatus of the present invention is characterized by comprising: a requested informa- 
tion input means for inputting the requested information of a user; a mixture ratio estimating means for estimating a 
mixture ratio indicating a ratio in which a foreground component image having foreground object components consti- 
tuting a foreground object and a background component image having background object components constituting a 
background object of the Image data that is obtained by an image pickup element having a predetemiined number of 
pixels, each having a time Integration effect, and that is comprised of pixel values detemnined for every pixels in ac- 
cordance with an amount of light fonning an image integrated over time are mixed on the basis of the requested 
Infomiation; a separating means for separating the image data into the foreground component image and the back- 
ground component Image on the basis of the mixture ratio; a synthesizing means for synthesizing the foreground 
component image separated by the separating means and any background component image, or the background 
component image separated by the separating means and any foreground component image in a predetemiined mix- 
ture ratio and generating the synthesized image; and a output means for outputting the synthesized image synthesized 
by the synthesizing means. 

[0871] The requested information input means may input amount of motion blur adjustment for adding hiotion blur 
to the foreground component image in addition to the requested infomiation of the user and further comprise a motion 
blur adding means which adds motion blur corresponding to amount by which motion blur Is adjusted to the foreground 
component image, and the synthesizing means may synthesize the foreground component image in which motion blur 
is added by the motion blur adding means and any background component image in a predetermined mixture ratio 
and generate the synthesized image. 

[0872] The requested infomnation Input means may also Input any background component Image in addition to the 
requested infomiation of the user, and the synthesizing means may synthesize the foreground component image and 
any background component image in a ratio estimated by the mixture ratio estimating means and generate the syn- 
thesized image. 

[0873] The charging means for performing charge processing in accordance with the requested infonmation may be 
further comprised. 

[0874] The charging means may generates charge infomiation including user ID, the communication apparatus ID 
and cost infomnation con^espondlng to the requested infomiation In accordance with the requested infonmation. 
[0875] The charging means may perform the charge processing with respect to a financial account of the user on 
the basis of the charge infomnation. 

[0876] The charging means may perform the charge processing by deducting the number of points corresponding 

to the cost infomnation from the number of points of each user used in the charge processing. 

[0877] The output means may output the synthesized image to the communication apparatus of the user having 

finished the charge processing after the charge processing has been finished by the charging means, 

[0878] The requested information Input means may input a predetemiined image data in addition to the requested 
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mfomiation of the user, and the output means may output the image synthesized by the synthesizing means to the 
communication apparatus of the user having finished the charge processing after the charge processing has been 
finished by the charging means. 

[0879] A 7*h communication method of the present invention Is characterized by comprising: a requested Information 
input step of Inputting the requested Infomiation of a user; a mixture ratio estimating step of estimating a mixture ratio 
indicating a ratio In which a foreground component Image having foreground object components constituting a fore- 
ground object and a background component image having baclcground object components constituting a background 
object of the image data that is obtained by an image pickup element having a predetennined number of pixels, each 
having a time integration effect, and that is comprised of pixel values detemiined for every pixels in accordance with 
an amount of light fomriing an image Integrated over time are mixed on the basis of the requested infomfiation; a sep- 
arating step of separating the image data Into the foreground component image and the background component image 
on the basis of the mixture ratio; a synthesizing step of synthesizing the foreground component Image separated In 
the separating step and any background component Image, or the background component Image separated in the 
separating step and any foreground component Image In a predetennined mixture ratio and generating the synthesized 
Image; and an output step of outputting the synthesized image synthesized in the synthesizing step. 
[0880] A program of a 10* recording medium of the present Invention is characterized by comprising: a requested 
infomiation input control step of controlling input of the requested Infomnation of a user; a mixture ratio estimation 
control step of controlling estimation of a mixture ratio Indicating a ratio in which a foreground component Image having 
foreground object components constituting a foreground object and a background component image having back- 
ground object components constituting a background object of the Image data that Is obtained by an image ptekup 
element having a predetennined number of pixels, each having a time integration effect, and that is comprised of pixel 
values detennined for every pixels in accordance with an amount of light fonning an image integrated over time are 
mixed on the basis of the requested infonnation; a separation control step of controlling separation of the image data 
into the foreground component image and the background component image on the basis of the mixture ratio; a syn- 
thesization control step of controlling synthesizatlon of the foreground component Image separated In the separation 
control step and any background component image, or the background component image separated In the separation 
control step and any foreground component image in a predetemnined mixture ratio and generating the synthesized 
image; and a output control step of controlling output of the synthesized image synthesized in the synthesizatlon control 
step. 

[0881] A 10th program of the present invention is characterized by making a computer perform: a requested infor- 
mation input control step of controlling input of the requested infonnation of a user; a mixture ratio estimation control 
step of controlling estimation of a mixture ratio Indicating a ratio in which a foreground component image having fore- 
ground object components constituting a foreground object and a background component image having background 
object components constituting a background object of the image data that is obtained by an image pickup element 
having a predetennined number of pixels, each having a time Integration effect, and that Is comprised of pixel values 
determined for every pixels in accordance with an amount of light fonning an image integrated overtime are mixed on 
the basis of the requested infonnation; a separation control step of controlling separation of the image data Into the 
foreground cortjponent image and the background component image on the basts of the mixture ratio; a synthesizatlon 
control step of controlling synthesizatlon of the foreground component image separated In the separation control step 
and any background component image, or the background component image separated in the separation control step 
and any foreground component image in a predetennined mixture ratio and generating the synthesized Image; and a 
output control step of controlling output of the synthesized image synthesized In the synthesizatlon control step. 
[0882] A 4*h communication system of the present Invention Is characterized in that the first communication apparatus 
comprises: a requested infonnation Input means for Inputting request information of a user; a requested infonnation 
transmitting means for transmitting the requested infonnation which is input by the requested information input means 
to the second transmitting apparatus; and a receiving means for receiving the synthesized Image transmitted from the 
second communication apparatus, and the second communfeation apparatus comprises: a requested information re- 
ceiving means for receiving the requested Infonnation which Is transmitted from the first transmitting apparatus; a 
mixture ratio estimating means for estimating a mixture ratio Indicating a ratio In which a foreground component Image 
having foreground object components constituting a foreground object and a background component Image having 
background object components constituting a background object of the Image data that is obtained by an image pickup 
element having a predetennined number of pixels, each having a time integration effect, and that Is comprised of pixel 
values determined for every pixels in accordance with an amount of light fonning an Image Integrated over time are 
mixed on the basis of the requested information; a separating means for separating the image data Into the foreground 
component Image and the background component image on the basis of the mixture ratio; a synthesizing means for 
synthesizing the foreground component image separated by the separating means and any background component 
image, or the background component image separated by the separating means and any foreground component Image 
In a predetennined mixture ratio and generating the synthesized image; and a synthesized Image transmitting means 
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for transmitting the synthesized image synthesized by the synthesizing means to the first communication apparatus. 
[0883] The requested information input means may input amount of motion blur adjustment for adding motion blur 
to the foreground component image in addition to the requested infomiation of the user, the second communication 
apparatus may further comprise a motion blur adding means which adds motion blur corresponding to amount by which 
5 motion blur Is adjusted to the foreground component image, and the synthesizing means may synthesize the foreground 
component image where motion blur is added by the motion blur adding means and any background component image 
in a predetermined mixture ratio and generate the synthesized Image. 

[0884] The requested information input means may also input any background component image in addition to the 
requested information of the user, the synthesizing means may synthesize the foreground component image and any 
10 background component image in a mixture ratio estimated by the mixture ratio estimating means and generate the 
synthesized image. 

[0885] The second communication apparatus may further comprise the charging means for performing charge 
processing in accordance with the requested information. 

[0886] The charging means may generates charge Infomnation including user ID, the second communication appa- 
15 ratus ID and cost infonnation con^esponding to the requested Infomiation in accordance with the requested infonmation. 
[0887] The charging means may perform the charge processing with respect to a financial account of the user on 

the basis of the charge information. 

[0888] The charging means of the second communication apparatus may perfomi the charge processing by deduct- 
ing the number of points corresponding to the cost Infomnation from the number of point of each user used In the charge 
20 processing. 

[0889] The synthesized image transmitting means may transmit the synthesized image to the first communteation 
apparatus of the user having finished the charge processing after the charge processing has been finished by the 
charging means. 

[0890] The requested infomiation input means may input a predetemiined image data in addition to the requested 
25 Infonnation of the user, and the output means may output the Image synthesized by the synthesizing means to the first 
communication apparatus of the user having finished the charge processing after the charge processing has been 
finished by the charging means. 

[0891] A communication method of the 4* communication system of the present invention is characterized by a 
communication method of the first communication apparatus comprises: a requested infomiation input step of Inputting 

30 request infonnation of a user; a requested Information transmitting step of transmitting the requested information which 
is input in the requested infonnation input step to the second transmitting apparatus; and a receiving step of receiving 
the synthesized image transmitted from the second communication apparatus, and a communication method of the 
second communication apparatus comprises: a requested information receiving step of receiving the requested infor- 
mation which is transmitted from the first transmitting apparatus; a mixture ratio estimating step of estimating a mixture 

35 ratio Indicating a ratio In which a foreground component image having foreground object components constituting a 
foreground object and a background component image having background object components constituting a back- 
ground object of the image data that Is obtained by an image pickup element having a predetemiined number of pixels, 
each having a time integration effect, and that is comprised of pixel values determined for every pixels in accordance 
with an amount of light fomning an image integrated over time are mixed on the basis of the requested information; a 

40 separating step of separating the image data into the foreground component image and the background component 
image on the basis of the mixture ratio; a synthesizing step of synthesizing the foreground component image separated 
in the separating step and any background component Image, or the background component image separated in the 
separating step and any foreground component image In a predetemiined mixture ratio and generating the synthesized 
image; and a synthesized image transmitting step of transmitting the synthesized image synthesized In the synthesizing 

45 step to the first communication apparatus. 

[0892] A program of anU^^ recording medium of the present invention is characterized In that a program controHIng 
the first communication apparatus comprises: a requested infonnation input control step of controlling input of the 
requested information of a user; a requested infonnation transmission control step of controlling transmission of the 
requested infomiation which Is input in the requested information input control step to the second transmitting appa- 

50 ratus; and a reception control step of controlling reception of the synthesized image transmitted from the second com- 
munication apparatus, and a program controlling the second communication apparatus comprises: a requested Infor- 
mation reception control step of controlling reception of the requested infonnation which is transmitted from the first 
transmitting apparatus; a mixture ratio estimation control step of controlling estimation of a mixture ratio Indicating a 
ratio In which a foreground component Image having foreground object components constituting a foreground object 

55 and a background component image having background object components constituting a background object of the 
image data that is obtained by an image pickup element having a predetemiined number of pixels, each having a time 
Integration effect, and that is comprised of pixel values determined for every pixels in accordance with an amount of 
light fomning an Image Integrated over time are mixed on the basis of the requested information; a separation control 
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step of controlling separation of the Image data Into the foreground component Image and the background component 
Image on the basis of the mixture ratio; a synthesization control step of controlling synthesization of the foreground 
component Image separated In the separation control step and any background component Image, or the background 
component Image separated In the separation control step and any foreground component image in a predetenmlned 
mixture ratio and generating the synthesized Image; and a synthesized Image transmission control step of controlling 
transmission of the synthesized image synthesized in the synthesization control step to the first communication appa- 
ratus. 

[0893] An 11 ^ program of the present Invention Is characterized by making a computer for controlling the first com- 
munication apparatus perform: a requested Infomnation input control step of controlling Input of the requested Infor- 
mation of a user; a requested Infomnation transmission control step of controlling transmission of the requested infor- 
mation whk^h is Input In the requested Infonnatlon input control step to the second transmitting apparatus; and a re- 
ception control step of controlling reception of the synthesized image transmitted from the second communication 
apparatus, and by making a computer for controlling the second communrcation apparatus perform: a requested in- 
formation reception control step of controlling reception of the requested infomriation which is transmitted from the first 
transmitting apparatus; a mixture ratio estimation control step of controlling estimation of a mixture ratio Indicating a 
ratio In which a foreground component image having foreground object components constituting a foreground object 
and a background component image having background object components constituting a background object of the 
image data that Is obtained by an image pickup element having a predetemnlned number of pixels, each having a time 
integration effect, and that Is comprised of pixel values detennined for every pixels in accordance with an amount of 
light fonning an image integrated over time are mixed on the basis of the requested infomnation; a separation control 
step of controlling separation of the image data Into the foreground component Image and the background component 
image on the basis of the mixture ratio; a synthesization control step of controlling synthesization of the foreground 
component Image separated In the separation control step and any background component image, or the background 
component Image separated In the separation control step and any foreground component Image In a predetermined 
mixture ratio and generating the synthesized Image; and a synthesized Image transmission control step of controlling 
transmission of the synthesized Image synthesized In the synthesization control step to the first communication appa- 
ratus. 

[0894] An 8*^ communication apparatus of the present invention is characterized by comprising: a requested infor- 
mation input means for inputting request information of a user; a requested infonnatlon transmitting means for trans- 
mitting the requested infomiation which is input by the requested infomnation input means to other communication 
apparatus; and a receiving means for receiving the synthesized image transmitted from other communication appara- 
tus. 

[0895] An 8*^ communication method of the present invention Is characterized by comprising: a requested infonnatlon 
Input step of inputting request Information of a user; a requested infomnation transmitting step of transmitting the re- 
quested information which is input in the requested information Input step to other communication apparatus; and a 
receiving step of receiving the synthesized image transmitted from other communication apparatus. 
[0896] A program of a 12^ recording medium of the present invention is characterized by comprising: a requested 
infonnatlon input control step of controlling input of request infonnatlon of a user; a requested infonnatlon transmission 
control step of controlling transmission of the requested Information which is Input In the requested Infonnatlon Input 
control step to other communication apparatus; and a reception control step of controlling reception of the synthesized 
Image transmitted from other communication apparatus. 

[0897] A 12^^ program of the present invention is characterized by making a computer perform: a requested infor- 
mation input control step of controlling input of request information of a user; a requested information transmission 
control step of controlling transmission of the requested information which Is input in the requested infomnation input 
control step to other communication apparatus; and a reception control step of controlling reception of the synthesized 
image transmitted from other communication apparatus. 

[0898] A 9*»i communication apparatus of the present Invention is characterized by comprising: a requested infomna- 
tion receiving means for receiving the requested Infonnatlon which Is transmitted from other communication apparatus; 
a mixture ratio estimating means for estimating a mixture ratio indicating a ratio in which a foreground component 
Image having foreground object components constituting a foreground object and a background component image 
having background object components constituting a background object of the image data that is obtained by an image 
pickup element having a predetemnlned number of pixels, each having a time Integration effect, and that Is comprised 
of pixel values detennined for every pixels In accordance with an amount of light forming an Image Integrated over 
time are mixed on the basis of the requested information; a separating means for separating the Image data into the 
foreground component image and the background component Image on the basis of the mixture ratio; a synthesizing 
means for synthesizing the foreground component image separated by the separating means and any background 
component Image, or the background component image separated by the separating means and any foreground com- 
ponent Image In a predetermined mixture ratio and generating the synthesized Image; and a synthesized image trans- 



83 



EP1 400 927 A1 



mitting means for transmitting the synthesized Image synthesized by the synthesizing means to other communication 

apparatus. 

I0899J A communication method of the present invention is characterized by comprising: a requested Information 
receiving step of receiving the requested infomriatlon which is transmitted from other communication apparatus; a 
mixture ratio estimating step of estimating a mixture ratio indicating a ratio in which a foreground component image 
having foreground object components constituting a foreground object and a bacl<ground component Image having 
background object components constituting a bacl<ground object of the Image data that is obtained by an image pickup 
element having a predetermined number of pixels, each having a time Integration effect, and that is comprised of pixel 
values detemilned for every pixels in accordance with an amount of light forming an image integrated over time are 
mixed on the basis of the requested infomiation; a separating step of separating the image data into the foreground 
component image and the background component image on the basis of the mixture ratio; a synthesizing step of 
synthesizing the foreground component image separated In the separating step and any background component image, 
or the background component image separated in the separating step and any foreground component Image In a 
predetermined mixture ratio and generating the synthesized Image; and a synthesized Image transmitting step of trans- 
mitting the synthesized Image synthesized in the synthesizing step to other communteation apparatus. 
[0900] A program of a 13^ recording medium of the present Invention is characterized by comprising: a requested 
Infomriatlon reception control step of controlling reception of the requested infonnation which Is transmitted from other 
communication apparatus; a mixture ratio estimation control step of controlling estimation of a mixture ratio Indicating 
a ratio In which a foreground component Image having foreground object components constituting a foreground object 
and a background component image having background object components constituting a background object of the 
image data that is obtained by an image pickup element having a predetermined number of pixels, each having a time 
integration effect, and that Is comprised of pixel values determined for every pixels In accordance with an amount of 
light fonming an Image Integrated over time are mixed on the basis of the requested information; a separation control 
step of controlling separation of the image data into the foreground component image and the background component 
image on the basis of the mixture ratio; a syntheslzation control step of controlling syntheslzation of the foreground 
component Image separated in the separation control step and any background component image, or the background 
component Image separated in the separation control step and any foreground component image in a predetermined 
mixture ratio and generating the synthesized image; and a synthesized image transmission control step of controlling 
transmission of the synthesized image synthesized in the syntheslzation control step to other communication appara- 
tus. 

[0901] A 1 3^ program of the present invention Is characterized by making a computer perfomi: a requested Infor- 
mation reception control step of controlling reception of the requested Infonnation which is transmitted from other 
communication apparatus; a mixture ratio estimation control step of controlling estimation of a mixture ratio indicating 
a ratio In which a foreground component image having foreground object components constituting a foreground object 
and a background component image having background object components constituting a background object of the 
image data that is obtained by an image pickup element having a predetermined number of pixels, each having a time 
integration effect, and that is comprised of pixel values detemnined for every pixels In accordance with an amount of 
light fonning an Image Integrated over time are mixed on the basis of the requested infonnation; a separation control 
step of controlling separation of the image data into the foreground component image and the background component 
image on the basis of the mixture ratio; a syntheslzation control step of controlling syntheslzation of the foreground 
component image separated In the separation control step and any background component image, or the background 
component Image separated in the separation control step and any foreground component image In a predetenmined 
mixture ratio and generating the synthesized image; and a synthesized Image transmission control step of controlling 
transmission of the synthesized image synthesized In the syntheslzation control step to other communication appara- 
tus. 

[0902] A communication apparatus of the present invention is characterized by comprising; an image ID input 
means for Inputting the ID discerning an Image requested by a user; a storing means for storing the Image having at 
least one image of a foreground component Image having foreground object components constituting a foreground 
object and a background component image having background object components constituting a background object 
separated from image data that is obtained by an image pickup element having a predetemnined number of pixels, 
each having a time integration effect, and that is comprised of pixel values detemnined for every pixels in accordance 
with an amount of light fonning an Image Integrated over time; a retrieving means for retrieving an Image corresponding 
to an image ID input by the image ID Input means of the Images stored In the storing means; an output means for 
outputting the Image retrieved by the retrieving means to the communication apparatus of the user; and a charging 
means for perfomiing charge processing in accordance with the Image retrieved by the retrieving means. 
[0903] A separating means may be further comprised for separating the image into a foreground component image 
having foreground object components constituting a foreground object and a background component image having 
background object components constituting a background object and the storing means may store the foreground 
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component image and the background component image separated from the image by the separating means. 
[0904] A synthesized Image generating means may be further comprised for synthesizing the foreground component 
image having foreground object components constituting a foreground object and the background component image 
having background object components constituting a background object separated from the image in a predetemiined 
mixture ratio and generating the synthesized image and the storing means may store the synthesized image generated 
by the synthesized image generating means. 

[0905] A separating means for separating the image into the foreground component image having foreground object 
components constituting a foreground object and the background component image having background object com- 
ponents constituting a background object and a synthesized image generating means for synthesizing the foreground 
component Image and the background component image separated by the separating means in a predetermined mix- 
ture ratio and generating the synthesized image may be further comprised, and the storing means may store the 
synthesized image generated by the synthesized image generating means. 

[0906] The charging means may generates charge information including user ID, the communication apparatus ID 
and cost information corresponding to the image retrieved by the retrieving means In accordance with the image re- 
trieved by the retrieving means. 

[0907] The charging means may perfomn the charge processing with respect to a financial account of the user on 
the basis of the charge information. 

[0908] The charging means may perfomri the charge processing by deducting the number of points corresponding 
to the cost infomriation from the number of point of each user used in the charge processing. 
[0909] The output means may output the synthesized image to the first communication apparatus of the user having 
finished the charge processing after the charge processing has been finished by the charging means, 
[0910] A 1 0^ communication method of the present invention is characterized by comprising; an image ID input step 
of inputting the ID discerning an image requested by a user; a storing step of storing the image comprised of at least 
one image of a foreground component image having foreground object components constituting a foreground object 
and a background component image having background object components constituting a background object sepa- 
rated from image data that is obtained by an image pickup element having a predetermined number of pixels, each 
having a time integration effect, and that is comprised of pixel values determined for every pixels in accordance with 
an amount of light fonming an image integrated over time; a retrieving step of retrieving an image corresponding to an 
image ID input in the image ID input step of the Images stored in the storing step; an output step of outputting the 
image retrieved in the retrieving step to the communication apparatus of the user; and a charging step of perfomiing 
charge processing according to the image retrieved in the retrieving step. 

[0911] A program of a 14*^ recording medium of the present invention Is characterized by comprising; an Image ID 
input control step of controlling input of the ID disceming an image requested by a user; a storage control step of 
controlling storage of the image comprised of at least one Image of a foreground component image having foreground 
object components constituting a foreground object and a background component image having background object 
components constituting a background object separated from image data that is obtained by an image pickup element 
having a predetemiined number of pixels, each having a time integration effect, and that is comprised of pixel values 
determined for every pixels in accordance with an amount of light forming an image integrated over time; a retrieval 
control step of controlling retrieval of an image corresponding to an image ID Input in the image ID input control step 
of the images stored in the storage control step; an output control step of controlling output of the image retrieved in 
the retrieval control step to the communication apparatus of the user; and a charge control step of control iing perform- 
ance of charge processing according to the image retrieved in the retrieval control step. 

[0912] A 14*^ program of the present invention is characterized by making a computer perfonn; an image ID input 
control step of controlling input of the ID discerning an image requested by a user; a storage control step of controlling 
storage of the image comprised of at least one image of a foreground component image having foreground object 
components constituting a foreground object and a background component Image having background object compo- 
nents constituting a background object separated from image data that is obtained by an image pickup element having 
a predetermined number of pixels, each having a time Integration effect, and that is comprised of pixel values deter- 
mined for every pixels in accordance with an amount of light forming an image integrated over time; a retrieval control 
step of controlling retrieval of an image corresponding to an image ID input in the Image ID input control step of the 
Images stored in the storage control step; an output control step of controlling output of the image retrieved in the 
retrieval control step to the communication apparatus of the user; and a charge control step of controlling perfomriance 
of charge processing according to the image retrieved in the retrieval control step. 

[0913] A 5*^^ communication system of the present invention is characterized in that a first communication apparatus 
comprises: an image ID input means for Inputting an ID discerning an Image requested by a user; an image ID trans- 
mitting means for transmitting the image ID to a second communication apparatus; and an image receiving means for 
receiving the image retrieved according to the image ID, and the second communication apparatus comprises: an 
image ID receiving means for receiving the image ID transmitted from the first communteation apparatus; a storing 
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means for storing the image comprised of at least one image of a foreground component image having foreground 
object components constituting a foreground object and a baclcground component image having bacicground object 
components constituting a background object separated from image data that is obtained by an image pickup eiement 
having a predetemfiined number of pixels, each having a time integration effect, and that is comprised of pixel values 
determined for every pixels in accordance with an amount of light fomriing an image Integrated over time; a retrieving 
means for retrieving the Image corresponding to the image ID received by the image ID receiving means among Images 
stored in the storing means; an output means for outputting the image retrieved by the retrieving means to the first 
communication apparatus; and a charging means for perfonning a charge processing according to the image retrieved 
by the retrieving means. 

[0914] The second communication apparatus may further comprise a separating means for separating the Image 
Into the foreground component Image having foreground object components constituting the foreground object and 
the background component image having background object components constituting the background object. 
[091 5] The storing means may store the foreground component image and the background component image sep- 
arated by the separating means. 

[091 6] The second communk:atlon apparatus may further comprise a synthesized Image generating means for syn- 
thesizing the foreground component image having foreground object components constituting the foreground object 
and the background component Image having background object components constituting the background object sep- 
arated from the image and generating the synthesized image in a predetemiined mixture ratio and the storing means 
may store the synthesized image generated by the synthesized image generating means. 

[0917] The second communication apparatus may further comprise: a separating means for separating the image 
into the foreground component image having foreground object components constituting the foreground object and 
the background component image having background object components constituting the background object; and a 
synthesized Image generating means for synthesizing the foreground component image having foreground object 
components constituting the foreground object and the background component image having background object com- 
ponents constituting the background object separated by the separating means and generating the synthesized image 
in a predetermined mixture ratio and the storing means may store the synthesized image generated by the synthesized 
Image generating means. 

[091 8] The charging means may generate charge information including a user ID, a second communication apparatus 
ID and cost infomnation corresponding to the Image retrieved by the retrieving means in accordance with the image 
retrieved by the retrieving means. 

[0919] The charging means may perform the charge processing with respect to a financial account of the user on 
the basis of the charge information. 

[0920] The charging means of the second communication apparatus may perform the charge processing by sub- 
tracting the number of points con^esponding to the cost information from the number of points of each user used In the 
charge processing. 

[0921] The output means may output the image to the first communteation apparatus of the user having finished the 
charge processing after the charge processing has been finished by the charging means. 

[0922] A communication method of the 5**^ communication system of the present invention is characterized In that a 
communication method of the first communication apparatus comprises: an Image ID Input step of Inputting an ID 
discerning an image requested by a user; an image ID transmitting step of transmitting the image ID to the second 
communication apparatus; and an image receiving step of receiving the image retrieved in accordance with the image 
ID, and a communication method of the second communication apparatus comprises: a storing step of storing the 
image comprised of at least one image of a foreground component image having foreground object components con- 
stituting a foreground object and a background component image having background object components constituting 
a background object separated from the image data that is obtained by an image pickup element having a predeter- 
mined number of pixels, each having a time Integration effect, and that Is comprised of pixel values detemiined for 
every pixels in accordance with an amount of light forming an image integrated over time; a retrieving step of retrieving 
the image corresponding to the image ID received in the image ID receiving step among images stored in the storing 
step; an output step of outputting the image retrieved in the retrieving step to the first communication apparatus; and 
a charging step of performing a charge processing in accordance with the image retrieved in the retrieving step. 
[0923] A program of a 15**^ recording medium of the present invention is characterized in that a program for controlling 
the first communication apparatus comprises: an image ID input control step of controlling Input of an ID discerning 
an image requested by a user; an Image ID transmission control step of controlling transmission of the image ID to the 
second communication apparatus; and an image reception control step of controlling reception of the image retrieved 
In accordance with the image ID, and a program for controlling the second communication apparatus comprises: a 
storage control step of controlling storage of the image comprised of at least one image of a foreground component 
image having foreground object components constituting a foreground object and a background component image 
having background object components constituting a background object separated from image data that is obtained 
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by an Image pickup element having a predetemiined number of pixels, each having a time Integration effect, and that 
Is comprised of pixel values determined for every pixels In accordance with an amount of light fonming an image Inte- 
grated over time; a retrieval control step of controlling retrieval of the Image corresponding to the image ID received 
In the Image ID reception control step among images stored In the storage control step; an output control step of 
controlling output of the image retrieved In the retrieval control step to the first communication apparatus; and a charge 
control step of controlling perfomiance of a charge processing In accordance with the Image retrieved In the retrieval 
control step. 

[0924] A 1 5^ program of the present invention is characterized by making a computer for controlling the first com- 
munication apparatus perfomn: an image ID input control step of controlling input of an ID discerning an image requested 
by a user; an image ID transmission control step of controlling transmission of the Image ID to the second communi- 
cation apparatus; and an image reception control step of controlling reception of the image retrieved in accordance 
with the Image ID, and by making a computer for controlling the second communication apparatus perfonm: a storage 
control step of controlling storage of the Image comprised of at least one Image of a foreground component Image 
having foreground object components constituting a foreground object and a background component image having 
background object components constituting a background object separated from image data that is obtained by an 
image pickup element having a predetemrilned number of pixels, each having a time Integration effect, and that Is 
comprised of pixel values detemiined for every pixels In accordance with an amount of light fomiing an image integrated 
over time; a retrieval control step of controlling retrieval of the image corresponding to the image ID received in the 
image ID reception control step among Images stored in the storage control step; an output control step of controlling 
output of the image retrieved in the retrieval control step to the first communication apparatus; and a charge control 
step of controlling perfomnance of a charge processing In accordance with the image retrieved in the retrieval control 
step. 

[0925] An 1 1th communication apparatus of the present Invention Is characterized by comprising: an image ID input 
means for inputting an ID discerning an image requested by a user; an image ID transmitting means for transmitting 
the image ID to other communication apparatus; and an image receiving means for receiving the image retrieved 
correspondingly to the image ID. 

[0926] An communication method of the present invention is characterized by comprising: an image ID input 
step of inputting an ID discerning an image requested by a user; an Image ID transmitting step of transmitting the Image 
ID to other communication apparatus; and an Image receiving step of receiving the image retrieved coaespondingly 
to the Image ID. 

[0927] A program of a 1 6*^ recording medium of the present invention is characterized by comprising: an image ID 
Input control step of controlling Input of an ID discerning an image requested by a user; an Image ID transmission 
control step of controlling transmission of the image ID to other communication apparatus; and an Image reception 
control step of controlling reception of the image retrieved correspondingly to the image ID. 

[0928] A 1 6* program of the present Invention Is characterized by making a computer perform: an Image ID Input 
control step of controlling input of an ID discerning an Image requested by a user; an image ID transmission control 
step of controlling transmission of the image ID to other communication apparatus; and an Image reception control 
step of controlling reception of the image retrieved correspondingly to the image ID. 

[0929] A 1 2* communteatlon apparatus of the present invention Is characterized by comprising: an Image ID receiving 
means for receiving an image ID transmitted from other communication apparatus; a storing means for storing an 
image comprised of at least one Image of a foreground component image having foreground object components con- 
stituting a foreground object and a background component image having background object components constituting 
a background object separated from image data that Is obtained by an Image pickup element having a predetemnined 
number of pixels, each having a time integration effect, and that Is comprised of pixel values detemnlned for every 
pixels in accordance with an amount of light fonming an image integrated over time; a retrieving means for retrieving 
the Image corresponding to the image ID received by the Image ID receiving means among images stored In the storing 
means; an output means for outputling the image retrieved by the retrieving means to other communication apparatus; 
and a charging means for performing a charge processing In accordance with the image retrieved by the retrieving 
means, A 12^^ communication method of the present Invention Is characterized by comprising: an image ID receiving 
step of receiving an Image ID transmitted from other communication apparatus; a storing step of storing an image 
comprised of at least one Image of a foreground component image having foreground object components constituting 
a foreground object and a background component image having background object components constituting a back- 
ground object separated from image data that is obtained by an image pickup element having a predetemnined number 
of pixels, each having a time integration effect, and that is comprised of pixel values determined for every pixels in 
accordance with an amount of light fonning an image Integrated over time; a retrieving step of retrieving the image 
corresponding to the image ID received In the image ID receiving step among Images stored in the storing step; an 
output step of outputting the image retrieved in the retrieving step to other communication apparatus; and a charging 
step of perfonning a charge processing in accordance with the Image retrieved In the retrieving step. 



87 



EP 1 400 927 A1 



[0930] A program of a 1 7^ recording medium of the present invention is characterized by comprising: an image ID 
reception control step of controlling reception of an image ID transmitted from other communication apparatus; a stor- 
age control step of controlling storage of an image comprised of at least one image of a foreground component image 
having foreground object components constituting a foreground object and a background component image having 
background object components constituting a background object separated from image data that is obtained by an 
image pickup element having a predetennlned number of pixels, each having a time integration effect, and that is 
comprised of pixel values detenmined for every pixels In accordance with an amount of light fomning an image integrated 
over time; a retrieval control step of controlling retrieval of the image corresponding to the Image ID received In the 
image ID reception control step among images stored in the storage control step; an output control step of controlling 
output of the image retrieved in the retrieval control step to other communication apparatus; and a charge control step 
of controlling performance of a charge processing in accordance with the Image retrieved in the retrieval control step. 
[0931] A 17* program of the present invention ischaracterizedby making a computer perform: an Image ID reception 
control step of controlling reception of an Image ID transmitted from other communication apparatus; a storage control 
step of controlling storage of an image comprised of at least one image of a foreground component image having 
foreground object components constituting a foreground object and a background component image having back- 
ground object components constituting a background object separated from image data that is obtained by an image 
pickup element having a predetemnined number of pixels, each having a time Integration effect, and that Is comprised 
of pixel values detemriined for every pixels in accordance with an amount of light fomiing an image Integrated over 
time; a retrieval control step of controlling retrieval of the Image corresponding to the Image ID received in the Image 
ID reception control step among Images stored in the storage control step; an output control step of controlling output 
of the Image retrieved in the retrieval control step to other communication apparatus; and a charge control step of 
controlling perfomiance of a charge processing In accordance with the image retrieved in the retrieval control step. 
[0932] A 13^1 communication apparatus of the present Invention is characterized by comprising: an input means for 
Inputting an Image comprised of at least one Image of a foreground component Image having foreground object com- 
ponents constituting a foreground object and a background component Image having background object components 
constituting a background object separated from Image data that is obtained by an Image pickup element having a 
predetermined number of pixels, each having a time integration effect, and that is comprised of pixel values determined 
for every pixels in accordance with an amount of light forming an image Integrated overtime; a storing means for storing 
the Image input by the input means; and a paying means for performing a payment processing In accordance with the 
Image stored In the storing means. 

[0933] A separating means for separating the image Into the foreground component Image having foreground object 
components constituting the foreground object and the background component image having background object com-- 
ponents constituting the background object may be further comprised and the input means may input the foreground 
component Image and the background component image separated by the separating means. 
[0934] A synthesized image generating means for synthesizing In a predetemnined mixture ratio the foreground com- 
ponent image having foreground object components constituting the foreground object and the background component 
image having background object components constituting the background object separated from the Image and gen- 
erating a synthesized Image may be further comprised and the Input means may input the synthesized image generated 
by the synthesized Image generating means. 

[0935] A separating means for separating the image into the foreground component Image having foreground object 
components constituting the foreground object and the background component image having background object com- 
ponents constituting the background object and a synthesized Image generating means for synthesizing In a prede- 
temnined mixture ratio the foreground component image and the background component Image separated by the sep- 
arating means and generating a synthesized Image may be further comprised and the Input means may Input the 
synthesized Image generated by the synthesized Image generating means. 

[0936] A 13* communication method of the present invention Is characterized by comprising: an Input step of Inputting 
an image comprised of at least one Image of a foreground component image having foreground object components 
constituting a foreground object and a background component Image having background object components consti- 
tuting a background object separated from image data that is obtained by an image pickup element having a prede- 
temnined number of pixels, each having a time Integration effect, and that Is comprised of pixel values determined for 
every pixels In accordance with an amount of light fomiing an image integrated over time; a storing step of storing the 
Image Input In the input step; and a paying step of perfonnlng a payment processing in accordance with the image 
stored In the storing step. 

[0937] A program of an 1 8*"^ recording medium of the present invention Is characterized by comprising: an Input 
control step of controlling input of an image comprised of at least one Image of a foreground component Image having 
foreground object components constituting a foreground object and a background component image having back- 
ground object components constituting a background object separated from Image data that is obtained by an image 
pickup element having a predetemnined number of pixels, each having a time integration effect, and that is comprised 
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of pixel values determined for every pixels In accordance with an amount of light forming an image Integrated over 
time; a storage control step of controlling storage of the image input in the input control step; and a payment control 
step of controlling performance of a payment processing In accordance with the image stored In the storage control step. 
[0938] A 1 8^ program of the present Invention is characterized by making a computer perfonn: an input control step 
of controlling Input of an image comprised of at least one image of a foreground component image having foreground 
object components constituting a foreground object and a background component image having background object 
components constituting a background object separated from image data that is obtained by an image pickup element 
having a predetermined number of pixels, each having a time integration effect, and that is comprised of pixel values 
detemnined for every pixels in accordance with an amount of light forming an image integrated over time; a storage 
control step of controlling storage of the image input in the input control step; and a payment control step of controlling 
performance of a payment processing In accordance with the Image stored in the storage control step. 
[0939] A communication system of the present invention is characterized In that a first communication apparatus 
comprises: an input means for inputting an image comprised of at least one image of a foreground component image 
having foreground object components constituting a foreground object and a background component image having 
background object components constituting a background object separated from image data that is obtained by an 
image pickup element having a predetennlned number of pixels, each having a time Integration effect, and that Is 
comprised of pixel values detemnined for every pixels in accordance with an amount of light fomnlng an image integrated 
overtime; and an image transmitting means for transmitting the image input by the input means to a second commu- 
nication apparatus, and the second communication apparatus comprises: an image receiving means for receiving the 
image transmitted from the first communication apparatus; a storing means for storing the image received by the image 
receiving means; and a paying means for perfonning a payment processing In accordance with the image stored in 
the storing means. 

[0940] The first communication apparatus may further comprise a separating means for separating the image into 
the foreground component image having foreground object components constituting the foreground object and the 
background component image having background object components constituting the background object and the input 
means may input the foreground component image and the background component Image separated by the separating 
means. 

[0941] The first communication apparatus may further comprise a synthesized image generating means for synthe- 
sizing in a predetennined mixture ratio the foreground component image having foreground object components con- 
stituting the foreground object and the background component image having background object components consti- 
tuting the background object separated from the image and generating a synthesized image and the input means may 
input the synthesized image generated by the synthesized Image generating means. 

[0942] The first communication apparatus may further comprise a separating means for separating the Image into 
the foreground component image having foreground object components constituting the foreground object and the 
background component image having background object components constituting the background object and a syn- 
thesized image generating means for synthesizing in a predetermined mixture ratio the foreground component image 
and the background component image separated by the separating means and generating a synthesized Image and 
the input means may input the synthesized image generated by the synthesized image generating means. 
[0943] A communication method of the 6*^ communication system is characterized in that a communication method 
of the first communication apparatus comprises: an input step of Inputting an Image comprised of at least one image 
of a foreground component image having foreground object components constituting a foreground object and a back- 
ground component image having background object components constituting a background object separated from 
image data that Is obtained by an image pickup element having a predetermined number of pixels, each having a time 
integration effect, and that is comprised of pixel values detemiined for every pixels in accordance with an amount of 
light forming an image integrated over time; and an image transmitting step of transmitting the Image input in the input 
step to a second communication apparatus, and a communication method of the second communication apparatus 
comprises: an image receiving step of receiving the image transmitted from the first communication apparatus; a storing 
step of storing the image received In the image receiving step; and a paying step of perfonming a payment processing 
in accordance with the image stored in the storing step. 

[0944] A program of the 1 9^ recording medium of the present invention is characterized in that a program for con- 
trolling the first communication apparatus comprises: an input control step of controlling input of an image comprised 
of at least one Image of a foreground component image having foreground object components constituting a foreground 
object and a background component Image having background object components constituting a background object 
separated from image data that is obtained by an image ptekup element having a predetennined number of pixels, 
each having a time integration effect, and that is comprised of pixel values determined for every pixels In accordance 
with an amount of light fomiing an Image Integrated over time; and an image transmission control step of controlling 
transmission of the image input in the Input control step to a second communication apparatus, and a program for 
controlling the second communication apparatus comprises: an image reception control step of controlling reception 
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of the image transmitted from the first communication apparatus; a storage control step of controlling storage of the 
image received in the image reception control step; and a payment control step of controlling performance of a payment 
processing in accordance with the image stored in the storage control step. 

[0945] A 1 9*^» program of the present Invention is characterized by making a computer for controlling a first commu- 
nication apparatus perfomn: an input control step of controlling input of an image comprised of at least one image of a 
foreground component image having foreground object components constituting a foreground object and a background 
component image having background object components constituting a background object separated from image data 
that is obtained by an image pickup element having a predetemiined number of pixels, each having a time integration 
effect, and that is comprised of pixel values determined for every pixels in accordance with an amount of light forming 
an image integrated over time; and an image transmission control step of controlling transmission of the image input 
In the Input control step to a second communication apparatus, and by making a computer for controlling the second 
communication apparatus perfomi: an image reception control step of controlling reception of the Image transmitted 
from the first communication apparatus; a storage control step of controlling storage of the image received in the image 
reception control step; and a payment control step of controlling performance of a payment processing in accordance 
with the image stored in the storage control step. 

[0946] A I4W1 communteatlon apparatus of the present Invention is characterized by comprising: an input means for 
inputting an image comprised of at least one image of a foreground component Image having foreground object com- 
ponents constituting a foreground object and a background component image having background object components 
constituting a background object separated from image data that is obtained by an image pickup element having a 
predetermined number of pixels, each having a time integration effect, and that is comprised of pixel values determined 
for every pixels in accordance with an amount of light fomning an image integrated overtime; and an Image transmitting 
means for transmitting the image input by the input means to other communication apparatus. 
[0947] A 1 4^ communteatlon method of the present invention is characterized by comprising: an input step of inputting 
an Image comprised of at least one image of a foreground component Image having foreground object components 
constituting a foreground object and a background component image having background object components consti- 
tuting a background object separated from Image data that is obtained by an image pickup element having a prede- 
termined number of pixels, each having a time integration effect, and that is comprised of pixel values detemnined for 
every pixels in accordance with an amount of light forming an image integrated over time; and an Image transmitting 
step of transmitting the image input in the input step to other communication apparatus. 

[0948] A program of a 20*^ recording medium of the present Invention is characterized by comprising : an Input control 
step of controlling input of an image comprised of at least one image of a foreground component image having fore- 
ground object components constituting a foreground object and a background component image having background 
object components constituting a background object separated from image data that is obtained by an image pickup 
element having a predetermined number of pixels, each having a time Integration effect, and that Is comprised of pixel 
values detemiined for every pixels In accordance with an amount of light fonning an image integrated over time; and 
an image transmission control step of controlling transmission of the image input in the input control step to other 
communication apparatus. 

[0949] A 20*^ program of the present invention is characterized by making a computer perfom: an input control step 
of controlling Input of an Image comprised of at least one Image of a foreground component image having foreground 
object components constituting a foreground object and a background component image having background object 
components constituting a background object separated from image data that is obtained by an image pickup element 
having a predetennined number of pixels, each having a time integration effect, and that is comprised of pixel values 
detemnined for every pixels in accordance with an amount of light fonning an image integrated overtime; and an image 
transmission control step of controlling transmission of the Image input In the input control step to other communteatlon 
apparatus. 

[0950] A 15^ communication apparatus of the present invention Is characterized by comprising: an Image receiving 
means for receiving an image transmitted from other communication apparatus; a storing means for storing the image 
received by the image receiving means; and a paying means for perfomnlng a payment processing in accordance with 
the image stored in the storing means. 

[0951] A 15*^ communication method of the present invention is characterized by comprising: an image receiving 
step of receiving an image transmitted from othercommunicatlon apparatus; a storing step of storing the image received 
in the image receiving step; and a paying step of performing a payment processing in accordance with the image stored 
in the storing step. 

[0952] A program of a 21** recording medium of the present invention is characterized by comprising: an image 
reception control step of controlling reception of an image transmitted from other communication apparatus; a storage 
control step of controlling storage of the image received In the image reception control step; and a payment control 
step of controlling perfonmance of a payment processing in accordance with the image stored in the storage control step. 
[0953] A 21«t program of the present invention Is characterized by making a computer perform: an Image reception 
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control step of controlling reception of an image transmitted from other communication apparatus; a storage control 
step of controlling storage of the image received in the image reception control step; and a payment control step of 
controlling perfomiance of a payment processing in accordance with the image stored in the storage control step. 
[0954] A 1 6^ communication apparatus of the present invention is characterized by comprising: a retrieval request 
Infomiation input means for inputting retrievai request infomnatlon of a user; a storing means for storing an image 
comprised of at least one image of a foreground component image having foreground object components constituting 
a foreground object and a background component image having background object components constituting a back- 
ground object separated from image data that is obtained by an image pickup element having a predetermined number 
of pixels, each having a time integration effect, and that is comprised of pixel values determined for every pixels in 
accordance with an amount of light fomning an Image integrated over time; a retrieving means for retrieving the image 
coH'espondingto the retrieval request infomiation input by the retrieval request infomiation input means among images 
stored in the storing means; an output means for outputting a retrieval result of the retrieving means; and a charging 
means for perfomriing a charge processing in accordance with the retrieval result. 

[0955] The storing means may comprise a plurality of Image storage units, respectively, provided with an image 
pickup element for picking up Image for updating image picked up by the image pickup element at a predetemnined 
time interval to store them and being capable of communication by a wire or wirelessly. 

[0956] The retrieval request information may include information specifying a foreground object, the storing means 
may further comprise a database for indicating relationship between an ID disceming the image pickup elements pro- 
vided in the plurality of Image storage units and an image object stored In the corresponding Image storage unit and 
the retrieving means may retrieve the image corresponding to the infonmatlon specifying the foreground object input 
by the retrieval request Infonmatlon input means among images stored in the storing means with reference to the 
database. 

[0957] A separating means for separating image data of the Image into the foreground component Image having 
foreground object components constituting the foreground object and the background component Image having back- 
ground object components constituting the background object may be further comprised and the storing means may 
store the foreground component Image and the background component Image separated by the separating means. 
[0958] A synthesized image generating means for synthesizing in a predetermined mixture ratio the foreground com- 
ponent Image having foreground object components constituting the foreground object and the background component 
image having background object components constituting the background object separated from the Image data of 
the image and generating a synthesized image may be further comprised and the storing means may store the syn- 
thesized image generated by the synthesized image generating means. 

[0959] A separating means for separating Image data of the image Into the foreground component Image having 
foreground object components constituting the foreground object and the background component Image having back- 
ground object components constituting the background object and a synthesized image generating means for synthe- 
sizing in a predetermined mixture ratio the foreground component image and the background component image sep- 
arated by the separating means and generating a synthesized image may be further comprised and the storing means 
may store the synthesized Image generated by the synthesized Image generating means. 

[0960] The output means may output existence and nonexistence of the image corresponding to the retrieval request 
Infomiation or the retrieved Image along with existence and nonexistence of the image con^esponding to the retrieval 
request infomiation as the retrieval result, and the charging means may perform the charge processing on the basis 
of the retrieval result output by the output means. 

[0961] The charging means may generate charge infomriation including a user ID, a communication apparatus ID 
and cost infomiation corresponding to the retrieval request infomiation In accordance with the retrieval request infor- 
mation. 

[0962] The charging means may perfonn the charge processing based on the retrieval result output by the output 

means with respect to a financial account of the user on the basis of the charge infomiation. 

[0963] The charging means may perfonn the charge processing by subtracting the number of points corresponding 

to the cost infomiation from the number of points of each user used for the charge processing. 

[0964] The output means may output the retrieval result to a communication apparatus of the user having finished 

the charge processing after the charge processing has been finished by the charging means. 

[0965] A 16*^ communication method of the present invention is characterized by comprising: a retrieval request 

infomiation input step of inputting retrieval request infomiation of a user; a storing step of storing an Image comprised 

of at least one Image of a foreground component Image having foreground object components constituting a foreground 

object and a background component image having background object components constituting a background object 

separated from image data that is obtained by an image pickup element having a predetemilned number of pixels, 

each having a time integration effect, and that is comprised of pixel values determined for every pixels In accordance 

with an amount of light fomiing an image integrated over time; a retrieving step of retrieving image corresponding to 

the retrieval request infomriation input in the retrieval request Infomiation input step among Images stored in the storing 
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step: an output step of outputting a retrieval result of the retrieving step; and a charging step of performing a charge 
processing in accordance with the retrievai resutt. 

[0966] A program of a 22"^ recording medium of the present invention is characterized by comprising: a retrieval 
request information input controi step of controiiing input of a retrieval request information of a user; a storage control 

5 step of controlling storage of an image comprised of at least one image of a foreground component image having 
foreground object components constituting a foreground object and a baclcground component image having back- 
ground object components constituting a baci<ground object separated from image data that is obtained by an image 
pickup element having a predetemiined number of pixels, each having a time integration effect, and that is comprised 
of pixei values detenmined for every pixels in accordance with an amount of light forming an image integrated over 

10 time; a retrieval control step of controlling retrieval of the image corresponding to the retrieval request Infomiation input 
in the retrieval request information input control step among images stored in the storage control step; an output control 
step of controlling output of a retrieval result of the retrievai control step; and a charge control step of controlling 
performance of a charge processing in accordance with the retrieval result. 

[0967] A 22"** program of the present invention is characterized by making a computer perform: a retrieval request 

15 infomiation input control step of controlling Input of a retrieval request Information of a user; a storage control step of 
controlling storage of an image comprised of at least one image of a foreground component image having foreground 
object components constituting a foreground object and a background component image having background object 
components constituting a background object separated from image data that is obtained by an image pickup element 
having a predetemnined number of pixels, each having a time Integration effect, and that is comprised of pixel values 

20 determined for every pixels in accordance with an amount of light fonnlng an image integrated over time; a retrieval 
control step of controlling retrieval of Image con^espondlng to the retrieval request infonmation Input in the retrieval 
request information Input control step among images stored in the storage control step; an output control step of con- 
trolling output of a retrieval result of the retrieval control step; and a charge control step of controlling perfonmance of 
a charge processing in accordance with the retrieval result. 

25 [0968] A 7*^ communication system of the present invention is characterized In that a first communication apparatus 
comprises: a retrieval request information input means for inputting a retrieval request information of a user; a retrieval 
request infomiation transmitting means for transmitting the retrieval request information to a second communication 
apparatus; and a retrieval result receiving means for receiving a retrieval result transmitted from the second commu- 
nication apparatus, and the second communication apparatus comprises: a retrieval request information receiving 

30 means for receiving the retrieval request information transmitted from the first communication apparatus; a storing 
means for storing an image comprised of at least one image of a foreground component image having foreground 
object components constituting a foreground object and a background component image having background object 
components constituting a background object separated from image data that is obtained by an image pickup element 
having a predetemnined number of pixels, each having a time integration effect, and that is comprised of pixel values 

35 detennined for every pixels In accordance with an amount of light fomriing an image integrated over time; a retrieving 
means for retrieving the Image corresponding to the retrieval request information received by the retrieval request 
Information receiving means among images stored In the storing means; an retrieval result transmitting means for 
transmitting the retrieval result of the retrieving means to the first communication apparatus; and a charging means 
for performing a charge processing In accordance with the retrieval result. 

40 [0969] The storing means may comprise a plurality of image storage units, respectively, provided with an image 
pickup element for picking up image for updating image picked up by the Image pickup element at a predetemiined 
time interval to store them and being capable of communteation by a wire or wirelessly. 

[0970] The retrieval request information may include infonmation specifying a foreground object, the storing means 
may further comprise a database for Indicating relationship between an ID discerning the Image pickup elements pro- 
45 vided in the plurality of Image storage units and an image object stored In the corresponding image storage unit and 
the retrieving means may retrieve the image corresponding to the infomiation specifying the foreground object input 
by the retrieval request information input means among Images stored In the storing means with reference to the 
database. 

[0971] The second communication apparatus may further comprise a separating means for separating Image data 
50 of the image Into the foreground component image having foreground object components constituting the foreground 
object and the background component image having background object components constituting the background object 
and the storing means may store the foreground component Image and the background component image separated 
by the separating means. 

[0972] The second communication apparatus may further comprise a synthesized Image generating means for syn- 
55 thesizing in a predetermined mixture ratio the foreground component image having foreground object components 
constituting the foreground object and the background component image having background object components con- 
stituting the background object separated from image data of the image and generating a synthesized image and the 
storing means may store the synthesized Image generated by the synthesized image generating means. 
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[0973] The second communication apparatus may further comprise a separating means for separating image data 
of the image into the foreground component image having foreground object components constituting the foreground 
object and the background component image having background object components constituting the background object 
and a synthesized image generating means for synthesizing in a predetermined mixture ratio the foreground component 

5 image and the background component image separated by the separating means and generating a synthesized image 
and the storing means may store the synthesized image generated by the synthesized image generating means. 
[0974] The output means may output existence and nonexistence of the image corresponding to the retrieval request 
infomiation or the retrieved image along with existence and nonexistence of the image con^esponding to the retrieval 
request information as the retrieval result and the charging means may perfonm the charge processing on the basis of 

10 the retrieval result output by the output means. 

[0975] The charging means may generate charge infonnation including a user ID, a communication apparatus ID 
and cost infonnation corresponding to the retrieval request Information in accordance with the retrieval request infor- 
mation. 

[0976] The charging means may perfomn the charge processing based on the retrieval result output by the output 

IS means with respect to a financial account of the user on the basis of the charge Infonmation. 

[0977] The charging means may perform the charge processing by subtracting the number of points corresponding 
to the cost infomfiation from the number of points of each user used for the charge processing. 
[0978] The output means may output the retrieval result to a communication apparatus of the user having finished 
the charge processing after the charge processing has been finished by the charging means. 

20 [0979] A communication method of the 7**^ communication system of the present invention is characterized in that a 
communication method of the first communication apparatus comprises: a retrieval request Information input step of 
inputting retrieval request infonnation of a user; a retrieval request infomnation transmitting step of transmitting the 
retrieval request infonnation to a second communication apparatus; and a retrieval result receiving step of recehfing 
a retrieval result transmitted from the second communication apparatus, and a communteatlon method of the second 

25 communication apparatus comprises: a retrieval request infonnation receiving step of receiving the retrieval request 
Information transmitted from the first communication apparatus; a storing step of storing an image comprised of at least 
one image of a foreground component Image having foreground object components constituting a foreground object 
and a background component image having background object components constituting a background object sepa- 
rated from image data that is obtained by an image pickup element having a predetemilned number of pixels, each 

30 having a time integration effect, and that is comprised of pixel values detemriined for every pixels in accordance with 
an amount of light forming an Image Integrated over time; a retrieving step of retrieving the Inriage corresponding to 
the retrieval request infonnation received in the retrieval request infonnation receiving step among images stored in 
the storing step; an retrieval result transmitting step of transmitting the retrieval result of the retrieving step to the first 
communication apparatus; and a charging step of perfonning a charge processing In accordance with the retrieval 

35 result. 

[0980] A program of a 23"* recording medium of the present invention is characterized in that a program for controlling 
the first communication apparatus comprises: a retrieval request infonnation input control step of controlling input of 
retrieval request Infonnation of a user; a retrieval request infonnation transmission control step of controlling transmis- 
sion of the retrieval request Infonnation to a second communication apparatus; and a retrieval result reception control 

40 step of controlling reception of a retrieval result transmitted from the second communication apparatus, and a program 
for controlling the second communication apparatus comprises: a retrieval request information reception control step 
of controlling reception of the retrieval request infonnation transmitted from the first communication apparatus; a storage 
control step of controlling storage of an image comprised of at least one image of a foreground component image 
having foreground object components constituting a foreground object and a background component Image having 

45 background object components constituting a background object separated from image data that is obtained by an 
image pickup element having a predetennlned number of pixels, each having a time integration effect, and that Is 
comprised of pixel values detemnlned for every pixels in accordance with an amount of light fonning an image integrated 
overtime; a retrieval control step of controlling retrieval of the image corresponding to the retrieval request infonnation 
received in the retrieval request information reception control step among images stored In the storage control step; 

so an retrieval result transmission control step of controlling transmission of the retrieval result of the retrieval control step 
to the first communication apparatus; and a charge control step of controlling performance of a charge processing In 
accordance with the retrieval result. 

[0981] A 23^ program of the present Invention is characterized by making a computer for controlling the first com- 
munication apparatus perfomn: a retrieval request infonnation input control step of controlling Input of retrieval request 
55 infonnation of a user; a retrieval request Infonnation transmission control step of controlling transmission of the retrieval 
request infonnation to a second communication apparatus; and a retrieval result reception control step of controlling 
reception of a retrieval result transmitted from the second communication apparatus, and by making a computer for 
controlling the second communication apparatus perfomn: a retrieval request infonnation reception control step of con- 
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light forming an image integrated over time; a retrievai control step of controiling retrieval of the image corresponding 
to the retrieval request infomiation received in the retrieval request infonnation reception control step among images 
stored in the storage control step; an retrieval result transmission control step of controlling transmission of a retrieval 
result of the retrieval control step to the other communication apparatus; and a charge control step of control perfonm- 
ance of a charge processing in accordance with the retrieval result. 

[0989] A 25^ program of the present invention is characterized by making a computer perfomi: a retrieval request 
infomnation reception control step of controlling reception of retrieval request infomiation transmitted from other com- 
munication apparatus; a storage control step of controlling storage of an image comprised of at least one Image of a 
foreground component image having foreground object components constituting a foreground object and a background 
component Image having background object components constituting a background object separated from image data 
that Is obtained by an image pickup element having a predetemnined number of pixels, each having a time integration 
effect, and that is comprised of pixel values determined for every pixels in accordance with an amount of light forming 
an image integrated over time; a retrieval control step of controlling retrievai of the image conrespondlng to the retrieval 
request information received in the retrieval request infomnation reception control step among images stored in the 
storage control step; an retrievai result transmission control step of controlling transmission of a retrieval result of the 
retrieval control step to the other communication apparatus; and a charge control step of control perfonmance of a 
charge processing in accordance with the retrieval result. 

[0990] A 19*^ communication apparatus of the present invention is characterized by comprising: a foreground com- 
ponent Image discerning information input means for inputting a foreground component image discerning information 
indicative of a foreground component image having foreground object components constituting a foreground object of 
image data that is obtained by an image pickup element having a predetermined number of pixels, each having a time 
integration effect, and that is comprised of pixel values determined for every pixels In accordance with an amount of 
light forming an Image integrated over time; a background component Image disceming Information input means for 
inputting a background component Image disceming Information indicative of a background component Image having 
background object components constituting a background object of the image data; an image positional infomiation 
generating means for generating a foreground component image positional infomiation and a background component 
image positional infomiation corresponding to the foreground component image disceming infomnation and the back- 
ground component image discerning infomnation; and an output means for outputting the foreground component image 
positional information and the background component image positional infomiation generated by the image positional 
infomiation generating means. 

[0991] A mixture ratio generating means for generating a mixture ratio of an area in which the foreground component 
image and the background component Image are mixed from the Image data that Is obtained by the image pickup 
element having the predetemnined number of pixels, each having a time integration effect, and that is comprised of 
pixel values detennined for every pixels in accordance with an amount of light fonning the image integrated over time, 
a separating means for separating the image data into the foreground component image and the background compo- 
nent image according to the mixture ratio, a discerning infomiation generating means for generating the foreground 
component image discerning infomiation based on the foreground component image and the background component 
image discerning information based on the background component image, respectively, and a discerning information 
supplying means for supplying the foreground component Image disceming infomnation generated by the discerning 
infomnation generating means to the foreground component image disceming infomnation input means and the back- 
ground component image discerning infomiation generated by the discerning infomiation generating means to the 
background component image discerning infomnation input means, respectively, may be further comprised. 
[0992] A storing means for storing at a predetemnined position the foreground component image and the background 
component image separated from the image data by the separating means and a positional infomnation storing means 
for storing each positional infomnation of the foreground component Image and the background component image of 
the image data stored by the storing means may be further comprised and the image positional infomnation generating 
means may generate the foreground component image positional infonnation and the background component image 
positional infomnation con-esponding to the foreground component image disceming infomnation and the background 
component image discerning information on the basis of the positional Infonmation stored by the positional information 
storing means and indicative of positions at which the foreground component image and the background component 
image are stored. 

[0993] The storing means may store the foreground component image and the background component image sep- 
arated from the image data by the separating means in other communication apparatus connected through a network, 
the positional information storing means may store each positional information on the network of the foreground com- 
ponent image and the background component image of the Image data stored In the other communication apparatus 
connected via the networt< by the storing means, and the image positional information generating means may generate 
the foreground component image positional information and the background component image positional information 
corresponding to the foreground component image disceming infomnation and the background component Image dis- 
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ceming information on the basis of the positional Information on the network of the foreground component image and 
the background component image of the image data stored by the posrtionaJ Information storing means. 
[0994] The storing means may store at a predetemnined position the foreground component image and the bacl<- 
ground component image separated from the Image data by the separating means along with IDs corresponding to 
the foreground component image and the background component image, respectively. 

[0995] A comparing means for comparing the respective IDs of the foreground component image and the background 
component image separated from the image data by the separating means with IDs of the foreground component 
image or the background component image previously stored by the storing means may be further comprised, and the 
storing means may store at a predetenmined position the foreground component Image and the background component 
image separated from the Image data by the separating means along with IDs corresponding to the foreground com- 
ponent Image and the background component image, respectively, on the basis of the comparison result of the com- 
paring means. 

[0996] When the comparison results of the comparing means are matched, the image positional information gener- 
ating means may generate the foreground component image positional infomnation and the background component 
image positional Infomnation corresponding to the foreground component Image disceming infomnation and the back- 
ground component image disceming Infomnation on the basis of the positional Infomnation of the foreground component 
image or the background component Image previously stored by the positional Information storing means. 
[0997] The foreground component Image discerning information input means may input the foreground component 
image having the foreground object components constituting the foreground object of the image data as the foreground 
component image disceming Information and the background component image discerning infomnation input means 
may Input the background component image having the background object components constituting the background 
object of the image data as the background component image disceming Infomnation. 

[0998] A charging means for perfomnlng a charge processing corresponding to output of the foreground component 
image positional information and the background component image positional Infomnation may be further comprised. 
[0999] The charging means may generate charge Information including a user ID, an ID of itself and cost infomnation 
corresponding to output of the foreground component Image positional information and the background component 

image positional information. 

[1000] The charging means may perfomn the charge processing with respect to a financial account of the user on 

the basis of the charge information. 

[1001] The charging means may perfonn the charge processing by subtracting the number of points corresponding 
to the cost infomnation from the number of points of each user used for the charge processing. 
[1002] The output means may output the foreground component image positional Infomnation and the background 
component image positional Infomnation after the charge processing has been finished by the charging means, In a 
way that only the user can obtain them. 

[1003] A 1 9* communication method of the present Invention Is characterized by comprising: a foreground compo- 
nent image discerning infomnation Input step of inputting a foreground component image discerning infomnation indic- 
ative of a foreground component Image having foreground object components constituting a foreground object of Image 
data that is obtained by an image pickup element having a predetermined number of pixels, each having a time inte- 
gration effect, and that Is comprised of pixel values detemnined for every pixels In acconJance with an amount of light 
forming an image integrated overtime; a background component Image discerning information Input step of Inputting 
a background component Image discerning infonmation indicative of a background component image having back- 
ground object components constituting a background object of the Image data; an image positional Information gen- 
erating step of generating a foreground component image positional infonmation and a background component Image 
positional infomnation conresponding to the foreground component image disceming Information and the background 
component image discerning information; and an output step of outputting the foreground component image positional 
Information and the background component Image positional Infomnation generated In the Image positional Infomnation 
generating step. 

[1 004] A program of a 26*^* recording medium of the present invention Is characterized by comprising: a foreground 
component image discerning Infonnation input control step of controlling input of a foreground component image dis- 
cerning Infomnation Indicative of a foreground component image having foreground object components constituting a 
foreground object of image data that Is obtained by an Image pickup element having a predetemnined number of pixels, 
each having a time integration effect, and that is comprised of pixel values detemnined for every pixels In accordance 
with an amount of light fonnnlng an Image Integrated over time; a background component Image disceming Infonnnatlon 
input control step of controlling input of a background component Image disceming infomnation Indicative of a back- 
ground component Image having background object components constituting a background object of the Image data; 
an image positional information generation control step of controlling generation of a foreground component image 
positional Information and a background component image positional Information corresponding to the foreground 
component Image discerning Infomnation and the background component image disceming Information; and an output 
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control step of controlling output of the foreground component image positional infomiation and the background com- 
ponent image positional information generated In the image positional infonnation generation control step. 
[1005] A 26**^ program of the present Invention is characterized by making a computer perfonn: a foreground com- 
ponent image discerning information Input control step of controlling input of a foreground component image discerning 
Information indicative of a foreground component image having foreground object components constituting a fore- 
ground object of image data that is obtained by an image pickup element having a predetermined number of pixels, 
each having a time Integration effect, and that is comprised of pixel values detennlned for every pixels In accordance 
with an amount of light fomning an Image Integrated over time; a background component image discerning infonnation 
input control step of controlling input of a background component Image discerning information indicative of a back- 
ground component image having background object components constituting a background object of the image data; 
an image positional Information generation control step of controlling generation of a foreground component image 
positional information and a background component image positional infomnation corresponding to the foreground 
component image discerning infonnation and the background component image disceming infonnation; and an output 
control step of controlling output of the foreground component image positional infonnation and the background com- 
ponent image positional infonnation generated In the image positional information generation control step. 
[1006] A 8*^ communication system of the present invention is characterized in that a first communication apparatus 
comprises: a foreground component image disceming infonnation input means for inputting a foreground component 
image disceming Infonnation indicative of a foreground component image having foreground object components con- 
stituting a foreground object of image data that is obtained by an image pickup element having a predetermined number 
of pixels, each having a time integration effect, and that is comprised of pixel values detennined for every pixels in 
accordance with an amount of light fonning an image Integrated over time; a background component image discerning 
information input means for inputting a background component image discerning information indicative of a background 
component image having background object components constituting a background object of the image data; a first 
transmitting means for transmitting the foreground component image disceming infonnation and the background com- 
ponent image discerning infonnation to a second communication apparatus; and a positional Information receiving 
means for receiving a foreground component image positional infonnation and a background component image posi- 
tional infonnation transmitted from the second communication apparatus, and the second communication apparatus 
comprises: a foreground component image discerning information receiving means for receiving the foreground com- 
ponent image discerning infonnation transmitted from the first communrcatlon apparatus; a background component 
Image discerning Information receiving means for receiving the background component image discerning Information 
transmitted from the first communication apparatus; an image positional infonnation generating means for generating 
a foreground component Image positional Information and a background component Image positional information cor- 
responding to the foreground component Image discerning information and the background component image discern- 
ing infonnation; and a second transmitting means for transmitting the foreground component image positional Infor- 
mation and the background component image positional information generated by the image positional Infonnation 
generating means to the first communication apparatus. 

[1007] The second communication apparatus may further comprise: a mixture ratio generating means for generating 
a mixture ratio of an area In which the foreground component image and the background component image are mixed 
from the Image data that Is obtained by the Image pickup element having the predetennlned number of pixels, each 
having a time integration effect, and that is comprised of pixel values detennined for every pixels in accordance with 
an amount of light forming the image integrated overtime; a separating means for separating the Image data Into the 
foreground component image and the background component image according to the mixture ratio; a disceming in- 
fonnation generating means for generating the foreground component image discerning Information on the basis of 
the foreground component image and the background component image discerning infonnation on the basis of the 
background component image, respectively; and a discerning Information supplying means for supplying the fore- 
ground component Image discerning Infonnation generated by the disceming infonnation generating means to the 
foreground component Image discerning infonnation input means and the background component Image discerning 
Infonnation generated by the discerning infonnation generating means to the background component Image disceming 
infonnation input means, respectively 

[1008] A storing means for storing at a predetemined position the foreground component image and the background 
component image separated from the Image data by the separating means and a positional infomnation storing means 
for storing each positional information of the foreground component image and the background component image of 
the image data stored by the storing means may be further comprised, and the image positional Infomnation generating 
means may generate the foreground component Image positional information and the background component image 
positional Infonnation con-esponding to the foreground component image disceming Infonnation and the background 
component image discerning information on the basis of the positional infonnation stored by the positional Infonnation 
storing means and indicative of positions at which the foreground component image and the background component 
image are stored. 
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[1 009] The storing means may store the foreground component Image and the background component image sep- 
arated from the image data by the separating means in other communication apparatus connected through a network, 
the positional Information storing means may store each positional infomiation on the networt< of the foreground com- 
ponent image and the background component image of the Image data stored in the other communication apparatus 
connected via the networt< by the storing means, and the image positional Infomiation generating means may generate 
the foreground component image positional infonmation and the background component image positional information 
corresponding to the foreground component Image disceming infonnation and the background component image dis- 
ceming infomiatlon on the basis of each positional infonnation on the network of the foreground component image and 
the background component image of the image data stored by the positional infonmation storing means. 
[1010] The storing means may store at a predetemiined position the foreground component Image and the back- 
ground component image separated from the image data by the separating means along with IDs corresponding to 
the foreground component Image and the background component image, respectively. 

[1011] The second commun cation apparatus may further comprise a comparing means for comparing the respective 
IDs of the foreground component image and the background component image separated from the image data by the 
separating means with IDs of the foreground component image or the background component image previously stored 
by the storing means, and the storing means may store at a predetemiined position the foreground component image 
and the background component image separated from the image data by the separating means along with IDs corre- 
sponding to the foreground component image and the background component image, respectively, on the basis of the 
comparison result of the comparing means. 

[1012] When the comparison results of the comparing means are matched, the image positional infonmation gener- 
ating means may generate the foreground component image positional infonnation and the background component 
image positional Infonnation conresponding to the foreground component image disceming infonnation and the back- 
ground component Image disceming infonnation on the basis of the positional Inf omiation of the foreground component 
image or the background component image previously stored by the positional Infonnation storing means. 
[1013] The foreground component Image discerning information input means may Input the foreground component 
image having the foreground object components constituting the foreground object of the image data as the foreground 
component image discerning information and the background component image discerning infomnation input means 
may Input the background component image having the background object components constituting the background 
object of the image data as the background component image discerning infonnation. 

[1014] The second communication apparatus may further comprise a charging means for performing a charge 
processing corresponding to output of the foreground component image positional infonnation and the background 
component Image positional infonnation. 

[1015] The charging means may generate charge infonnation including a user ID, an ID of the second communication 
apparatus and cost infonmation con-esponding to output of the foreground component Image positional infonnation and 
the background component Image positional infonnation. 

[1016] The charging means may perfonn the charge processing with respect to a financial account of the user on 
the basis of the charge Infonnation. 

[1017] The charging means may perfonn the charge processing by subtracting the number of points corresponding 
to the cost infonnation from the number of points of each user used for the charge processing. 
[1018] The output means may output the foreground component Image positional infonnation and the background 
component image positional infonmation to the user having finished the charge processing after the charge processing 
has been finished by the charging means. 

[1019] A communication method of the 8*^ communication system of the present invention is characterized in that a 
communication method of the first communication apparatus comprises: a foreground component image discerning 
information input step of inputting a foreground component image disceming infonnation indicative of a foreground 
component image having foreground object components constituting a foreground object of image data that is obtained 
by an image pickup element having a predetermined number of pixels, each having a time integration effect, and that 
is comprised of pixel values determined for every pixels in accordance with an amount of light fonning an Image inte- 
grated over time; a background component Image disceming information input step of inputting a background compo- 
nent image disceming infonnation Indicative of a background component image having background object components 
constituting a background object of the image data; a first transmitting step of transmitting the foreground component 
Image discerning infonnation and the background component image disceming information to a second communteation 
apparatus; and a positional infonnation receiving step of receiving a foreground component Image positional Infonna- 
tion and a background component image positional infonmation transmitted from the second communication apparatus, 
and a communication method of the second communication apparatus comprises: a foreground component image 
discerning information receiving step of receiving the foreground component image discerning Infonmation transmitted 
from the first communication apparatus; a background component image disceming infonnation receiving step of re- 
ceiving the background component image discerning infonnation transmitted from the first communication apparatus; 
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an image positional information generating step of generating the foreground component image positional information 
and the background component image positional information con-esponding to the foreground component image dis- 
cerning information and the bacl^ground component image discerning infomiation; and a second transmitting step of 
transmitting the foreground component Image positional infomnation and the background component image positional 
infonnation generated in the image positional infomnation generating step to the first communication system. 
[1 020] A program of a 27^ recording medium of the present invention is characterized in that a program for controlling 
the first communication apparatus comprises: a foreground component image disceming Infomiation input control step 
of controlling input of a foreground component Image disceming infomnation indtcath^e of a foreground component 
image having foreground object components constituting a foreground object of image data that is obtained by an 
image pickup element having a predetemiined number of pixels, each having a time Integration effect, and that is 
comprised of pixel values detemnined for every pixels in accordance with an amount of light fonming an Image Integrated 
over time; a background component image discerning Information Input control step of controlling input of a background 
component image discerning infomnation indicative of a background component image having background object com- 
ponents constituting a background object of the image data; a first transmission control step of controlling transmission 
of the foreground component image discerning infonnation and the background component image disceming infomna- 
tion to a second communication apparatus; and a positional infomnation reception control step of controlling reception 
of a foreground component image positional infonnation and a background component image positional information 
transmitted from the second communication apparatus, and a program for controlling the second communication ap- 
paratus comprises: a foreground component image discerning infomnation reception control step of controlling recep- 
tion of the foreground component image disceming information transmitted from the first communication apparatus; a 
background component image disceming infonnation reception control step of controlling reception of the background 
component image disceming information transmitted from the first communication apparatus; an image positional in- 
formation generation control step of controlling generation of the foreground component image positional infomnation 
and the background component image positional infonnation corresponding to the foreground component image dis- 
cerning infonnation and the background component image disceming infonnation; and a second transmission control 
step of controlling transmission of the foreground component image positional information and the background com- 
ponent image positional infonnation generated in the image positional information generation control step to the first 
communication system. 

[1 021 ] A 27^^ program of the present Invention is characterized by making a computer for controlling a first commu- 
nication apparatus perfomn: a foreground component image disceming infomnation Input control step of controlling 
input of a foreground component image discerning infomiation Indicative of a foreground component image having 
foreground object components constituting a foreground object of image data that is obtained by an image pickup 
element having a predetermined number of pixels, each having a time integration effect, and that is comprised of pixel 
values detemnined for every pixels in accordance with an amount of light fomning an image integrated over time; a 
background component Image discerning infonnation input control step of controlling input of a background component 
image discerning infonnation indicative of a background component image having background object components 
constituting a background object of the Image data; a first transmission control step of controlling transmission of the 
foreground component image discerning information and the background component image disceming information to 
a second communication apparatus; and a positional infonnation reception control step of controlling reception of a 
foreground component Image positional information and a background component image positional Infonnation trans- 
mitted from the second communication apparatus, and by making a computer for controlling the second communication 
apparatus perform: a foreground component Image disceming information reception control step of controlling recep- 
tion of the foreground component image disceming infonnation transmitted from the first communication apparatus; a 
background component image disceming infomnation reception control step of controlling reception of the background 
component image disceming infomnation transmitted from the first communication apparatus; an image positional in- 
formation generation control step of controlling generation of the foreground component image positional infonnation 
and the background component image positional information corresponding to the foreground component Image dis- 
cerning Infonnation and the background component image discerning Infonnation; and a second transmission control 
step of controlling transmission of the foreground component image positional information and the background com- 
ponent image positional information generated in the image positional infonnation generation control step to the first 
communication system. 

[1022] A 20^ communication apparatus of the present invention is characterized by comprising: a foreground com- 
ponent Image discerning infonnation input means for inputting a foreground component Image disceming infonnation 
indicative of a foreground component image having foreground object components constituting a foreground object of 
image data that is obtained by an image pickup element having a predetermined number of pixels, each having a time 
Integration effect, and that is comprised of pixel values determined for every pixels in accordance with an amount of 
light forming an image integrated over time; a background component image discerning Infonnation input means for 
inputting a background component Image disceming infomnation Indicative of a background component image having 
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background object components constituting a background object of the image data; a transmitting means for transmit- 
ting the foreground component image disceming infonmation and the background component image disceming infor- 
mation to other communteation apparatus through a network; and a positional infonnation receiving means for receiving 
a foreground component image positional infomnatlon and a background component image positional Information trans- 
mitted from the other communication apparatus through a network. 

[1023] A 20^ communteation method of the present invention Is characterized by comprising: a foreground compo- 
nent image disceming Infonnation input step of Inputting a foreground component image disceming information Indic- 
ative of a foreground component Image having foreground object components constituting a foreground object of image 
data that is obtained by an image pickup element having a predetermined number of pixels, each having a time inte- 
gration effect, and that is comprised of pixel values detemiined for every pixels In accordance with an amount of light 
forming an Image Integrated over time; a background component image discerning Infonnation Input step of inputting 
a background component image discerning Infonnation indicative of a background component image having back- 
ground object components constituting a background object of the image data; a transmitting step of transmitting the 
foreground component image disceming Infonnation and the background component Image disceming Information to 
other communication apparatus through a networi<; and a positional infonnation receiving step of receiving a foreground 
component image positional information and a background component image positional information transmitted from 
the other communication apparatus via the network. 

[1024] A program of a 28**^ recording medium of the present invention is characterized by comprising: a foreground 
component image discerning infonnation input control step of controlling input of a foreground component image dis- 
cerning information Indicative of a foreground component image having foreground object components constituting a 
foreground object of Image data that Is obtained by an Image pickup element having a predetennlned number of pixels, 
each having a time Integration effect, and that is comprised of pixel values detemnined for every pixels in accordance 
with an amount of light fomrilng an image integrated over time; a background component Image disceming infonnation 
input control step of controlling input of a background component image disceming information indicative of a back- 
ground component Image having background object components constituting a background object of the image data; 
a transmission control step of controlling transmission of the foreground component image discerning Information and 
the background component image discerning infonnation to other communication apparatus through a network; and 
a positional Infonnation reception control step of controlling reception of a foreground component Image positional 
information and a background component image positional Information transmitted from the other communication ap- 
paratus via the networic. 

[1025] A 28*^ program of the present invention is characterized by making a computer perform: a foreground com- 
ponent image disceming infonnation input control step of controlling input of a foreground component image discerning 
information indicative of a foreground component image having foreground object components constituting a fore- 
ground object of image data that Is obtained by an image pickup element having a predetemnined number of pixels, 
each having a time integration effect, and that is comprised of pixel values detennined for every pixels in accordance 
with an amount of light fonning an image integrated overtime; a background component Image discerning infonnation 
input control step of controlling input of a background component image disceming infonnation Indicative of a back- 
ground component Image having background object components constituting a background object of the image data; 
a transmission control step of controlling transmission of the foreground component Image disceming information and 
the background component image discerning information to other communication apparatus through a network; and 
a positional infonnation reception control step of controlling reception of a foreground component image positional 
information and a background component image positional Infonnation transmitted from the other communication ap- 
paratus via the network. 

[1 026] A 21 ^ communication apparatus of the present invention is characterized by comprising: a foreground com- 
ponent image disceming information receiving means for receiving the foreground component image discerning Infor- 
mation of image data transmitted from other communication apparatus through a network, the image data being ob- 
tained by an Image pickup element having a predetennlned number of pixels, each having a time integration effect, 
and that is comprised of pixel values detennined for every pixels in accordance with an amount of light fonning an 
image integrated overtime; a background component image discerning Infonnation receiving means for receiving the 
background component image disceming information transmitted from the other communication apparatus; an image 
positional Infonnation generating means for generating a foreground component image positional Infonnation and a 
background component image positional infonnation corresponding to the foreground component image discerning 
infonnation and the background component image disceming Infonnation; and a transmitting means for transmitting 
the foreground component image positional information and the background component Image positional information 
generated by the image positional Infonnation generating means to the other communication apparatus via the network. 
[1027] A 21 St communication method of the present invention is characterized by comprising: a foreground compo- 
nent image discerning information receiving step of receiving the foreground component image discerning infonnation 
of image data transmitted from other communication apparatus through a networic, the image data being obtained by 
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an image pickup element having a predetemiined number of pixels, each having a time integration effect, and that is 
comprised of pixel values detem^iined for every pixels in accordance with an amount of light forming an image integrated 
overtime; a background component image disceming Infomriation receiving step of receiving the background compo- 
nent image disceming Infomiatlon transmitted from the other commun nation apparatus; an Image positional Information 
generating step of generating a foreground component image positional infomiation and a background component 
Image positional Infomnatlon corresponding to the foreground component Image disceming infomiatlon and the back- 
ground component image disceming Infonnatlon; and a transmitting step of transmitting the foreground component 
image positional information and the background component image positional information generated in the image 
positional infomriation generating step to the other communication apparatus via the network. 
[1028] A program of a 29^ recording medium of the present Invention is characterized by comprising: a foreground 
component Image discerning Infonmatlon reception control step of controlling reception of the foreground component 
Image disceming Infomriation of image data transmitted from other communication apparatus through a network, the 
Image data being obtained by an Image prckup element having a predetermined number of pixels, each having a time 
Integration effect, and that Is comprised of pixel values detemnined for every pixels In accordance with an amount of 
light fonming an image integrated over time; a background component image disceming information reception control 
step of controlling reception of the background component image disceming infomriation transmitted from the other 
communication apparatus; an image positional infomriation generation control step of controlling generation of a fore- 
ground component image positional Information and a background component Image positional information corre- 
sponding to the foreground component Image disceming Information and the background component Image discerning 
Information; and a transmission control step of controlling transmission of the foreground component image positional 
infomriation and the background component image positional Infonnatlon generated In the image positional infonmation 
generation control step to the other communication system via the network. 

[1029] A 29^ program of the present invention is characterized by making a computer perform: a foreground com- 
ponent image disceming infonmation reception control step of controlling reception of the foreground component image 
disceming infomriation of image data transmitted from other communication apparatus through a network, the Image 
data being obtained by an image pickup element having a predetermined number of pixels, each having a time inte- 
gration effect, and that Is comprised of pixel values detemnined for every pixels In accordance with an amount of light 
fonning an Image integrated over time; a background component image discerning Information reception control step 
of controlling reception of the background component image discerning information transmitted from the other com- 
munication apparatus; an image positional infonnatlon generation control step of controlling generation of a foreground 
component image positional information and a background component image positional information corresponding to 
the foreground component Image discerning infonnatlon and the background component image discerning infonmation; 
and a transmission control step of controlling transmission of the foreground component Image positional Infonnatlon 
and the background component Image positional information generated In the image positional Infonmation generation 
control step to the other communication system via the network. 

[1030] In the 1^ communication system and method and the 2^ program of the present invention, by a first com- 
munication apparatus, requested information of a user is Input from among a plurality of encoded data which are 
generated on the basis of predetermined image data that is obtained by an image pickup element having a predeter- 
mined number of pixels, each having a time integration effect, and that is comprised of pixel values determined for 
every pixels In accordance with an amount of light fonning an (mage integrated overtime; the Input request Infonnatlon 
is transmitted to the second communication apparatus; and the encoded data generated on the basis of the Image 
data corresponding to the requested infonnation transmitted from the second communication apparatus Is received, 
and by the second communication apparatus, the requested information transmitted from the first communication ap- 
paratus is received; the image data is separated into a foreground component image having foreground object com- 
ponents constituting a foreground object and a background component image having background object components 
constituting a background object; and the encoded data generated when the Image is separated into the foreground 
component image and the background component image on the basis of the received request information is transmitted 
to the first communication apparatus. 

[1031] In the 2"^ communication apparatus and method and the 3"^ program of the present invention, the request 
requested infonnatlon of a user is input from among a plurality of encoded data which are generated on the basis of 
predetermined image data that is obtained by an image pickup element having a predetermined number of pixels, each 
having a time integration effect, and that is comprised of pixel values detennined for every pixels in accordance with 
an amount of light fonning an image Integrated over time; the input request information is transmitted to other commu- 
nication apparatus; and the encoded data generated on the basis of the image data corresponding to the requested 
infonnation, transmitted from other communication apparatus, Is received. 

[1032] In the 3"* communication apparatus and method and the 4^ program of the present invention, the requested 
infonnation transmitted from other communication apparatus is received; the image data that is obtained by an image 
pickup element having a predetermined number of pixels, each having a time integration effect, and that is comprised 
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of pixel values determined for every pixels in accordance with an amount of light fomiing an image integrated over 
time is separated into a foreground component Image having foreground object components constituting a foreground 
object and a background component image having background object components constituting a background object; 
and the encoded data generated when the image is separated into the foreground component Image and the back- 
ground component image on the basis of the received request Infonmation is transmitted to other communication ap- 
paratus. 

[1 033] in the 2^ communication system and method and the 5* program of the present Invention, by the first com- 
munteation apparatus, the request requested infomnation of a user Is input from among a plurality of encoded data 
whbh are generated on the basis of predetermined image data that is obtained by an Image ptekup element having a 
predetermined number of pixels, each having a time integration effect, and that is comprised of pixel values determined 
for every pixels in accordance with an amount of light fonnlng an Image integrated over time; the input request infor- 
mation Is transmitted to one of the second communication apparatus to the fourth communication apparatus; and the 
encoded data generated on the basis of the image data corresponding to the requested information, transmitted from 
the one of the second communication apparatus to the fourth communication apparatus, is received, by the second 
communication apparatus, the requested information transmitted from the first communteation apparatus Is received; 
the image data is separated into a foreground component image having foreground object components constituting a 
foreground object and a background component image having background object components constituting a back- 
ground object; and the foreground component image and the background component image separated from the image 
on the basis of the received request information are transmitted as the encoded data to the first communication appa- 
ratus, by the third communication apparatus, the requested information transmitted from the first communication ap- 
paratus is received; area infomiation discerning any of a foreground area having foreground object components con- 
stituting a foreground object of the image data, a background area having background object components constituting 
a background object of the predetermined image data and a mixed area (n which the foreground area and the back- 
ground area are mixed is generated; and the generated area infonnatibn based on the received request infomiation 
is transmitted as the encoded data to the first communication apparatus, and by the fourth communtoation apparatus, 
the requested Infonnatlon transmitted from the first communication apparatus is received; a mixture ratio Indicating a 
ratio in which the foreground area and the background area are mixed in the mixed area of the Image data is generated; 
and the mixture ratio generated on the basis of the received request Infonmation is transmitted to the first communbation 
apparatus. 

[1 034] In the 4^^ communteatlon apparatus and method and the 6*^ program of the present invention, the requested 
infomiation of a user is input; in accordance with the requested Information, a foreground component image having 
foreground object components constituting a foreground object and a background component image having back- 
ground object components constituting a background object of the image data of a predetennined image data that is 
obtained by an image pickup element having a predetermined number of pixels, each having a time Integration effect, 
and that is comprised of pixel values detemnined for every pixels in accordance with an amount of light forming an 
image integrated over time are synthesized and a synthesized Image is generated; and the generated synthesized 
image Is output, 

[1 035] In the 3^^ communication system and method and the 7*^ program of the present Invention, by the first com- 
munk:ation apparatus, the requested information of a user Is Input; the Input request information Is transmitted to the 
second communication apparatus; and in accordance with the requested infomiation, the synthesized Image transmit- 
ted from the second communication apparatus is input, and by the second communication apparatus, the requested 
infomnation transmitted from the first communication apparatus Is received; In accordance with the requested Informa- 
tion, a foreground component image having foreground object components constituting a foreground object and a 
background component image having background object components constituting a background object of predeter- 
mined image data that is obtained by an image pickup element having a predetermined number of pixels, each having 
a time integration effect, and that is comprised of pixel values detemnined for every pixels in accordance with an amount 
of light forming an image integrated over time are synthesized and a synthesized image is generated; and the generated 
synthesized image is transmitted to the first communication apparatus. 

[1036] In the 5*^ communication apparatus and method and the 8^ program of the present Invention, the requested 
Infonnatlon of a user Is input; the input request information is transmitted to other communication apparatus; and the 
synthesized data transmitted from other communication apparatus In accordance with the requested infonmation is 
received. 

[1 037] In the 6*h communteation apparatus and method and the 9*** program of the present Invention, the requested 
infonmation transmitted from other communication apparatus Is received; in accordance with the requested information, 
a foreground component image having foreground object components constituting a foreground object and a back- 
ground component image having background object components constituting a background object of predetermined 
Image data that is obtained by an image pickup element having a predetermined number of pixeis, each having a time 
integration effect, and that is comprised of pixel values determined for every pixels in accordance with an amount of 
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light forming an image integrated overtime are synthesized and the synthesized image is generated; and the generated 
synthesized image is transmitted to other communication apparatus. 

(1 038] In the 7^ communication apparatus and method and the 1 0**^ program of the present invention, the requested 
infomiation of a user Is Input; in accordance with the requested infomiation, a mixture ratio indicating a ratio In which 
a foreground component image having foreground object components constituting a foreground object to a background 
component image having bacl<ground object components constituting a baclcground object of the image data are mixed 
is estimated, the image data being is obtained by an image pickup element having a predetermined number of pixels, 
each having a time Integration effect, and that is comprised of pixel values detenmined for every pixels In accordance 
with an amount of light fomning an image integrated over time; on the basis of the mixture ratio, the image data is 
separated Into the foreground component image and the background component Image; the separated foreground 
component image and any background component image, or the separated background component image and any 
foreground component Image are synthesized In a predetemnined mixture ratio and a synthesized image Is generated; 
and the synthesized image is output. 

[1 0391 In the 4<h communfcation system and method and the 1 1 program of the present Invention, by the first com- 
munk:at{on apparatus, the requested information of a user ts input; the Input request infonmatlon is transmitted to the 
second transmitting apparatus; and the synthesized image transmitted from the second communication apparatus is 
received, and by the second communication apparatus, the requested information transmitted from the first commu- 
nication apparatus Is received; a mixture ratio Indicating a ratio In which a foreground component Image having fore- 
ground object components constituting a foreground object and a background component image having background 
object components constituting a background object of the image data are mixed is estimated on the basis of the 
requested infomiation, the image data being obtained by an image pickup element having a predetermined number of 
pixels, each having a time Integration effect, and that is comprised of pixel values determined for every pixels in ac* 
cordance with an amount of light forming an image Integrated overtime; the image data is separated Into the foreground 
component Image and the background component Image on the basis of the mixture ratio; the separated foreground 
component Image and any background component image or the separated background component image and any 
foreground component Image are synthesized In a predetermined mixture ratio and the synthesized image is generated; 
and the synthesized image is transmitted to the first communk^atlon apparatus. 

[1040] In the 8* communfcation apparatus and method and the 12^^ program of the present Invention, the requested 
information of a user is Input; the requested Infonnatlon is transmitted to other communication apparatus; and the 
synthesized image transmitted from other communication apparatus Is received. 

[1041] In the 9*^ communication apparatus and method and the 1 3*^ program of the present invention, the requested 
infomiation transmitted from other communication apparatus Is received; a mixture ratio indicating a ratio in which a 
foreground component image having foreground object components constituting a foreground object and a background 
component image having background object components constituting a background object of the Image data are mixed 
is estimated on the basis of the requested Infomiation, the image data being obtained by an Image pickup element 
having a predetermined number of pixels, each having a time integration effect, and that is comprised of pixel values 
determined for every pixels in accordance with an amount of light fomrilng an Image Integrated over time; the Image 
data is separated into the foreground component Image and the background component image on the basis of the 
mixture ratio; the separated foreground component Image and any background component image or the separated 
background component Image and any foreground component Image are synthesized in a predetermined ratio and 
the synthesized image is generated; and the synthesized image Is transmitted to other communication apparatus. 
[1042] In the 10**^ communication apparatus and method and the 14^^ program of the present invention, an ID dis- 
cerning an Image requested by a user is input; an Image comprised of at least one Image of a foreground component 
image having foreground object components constituting a foreground object and a background component Image 
having background object components constituting a background object separated from the image data that is obtained 
by an Image pickup element having a predetermined number of pixels, each having a time Integration effect, and that 
is comprised of pixel values determined for every pixels In accordance with an amount of light fonming an Image inte- 
grated over time, is stored; the image con-esponding to the input image ID is retrieved from among the stored Images; 
the retrieved image is transmitted to the communication apparatus of the user; and the charge processing is perf onned 
in accordance with the retrieved Image. 

[1043] In the 5*^ communication system and method and the 15*^ program of the present Invention, by the first com- 
munication apparatus, an ID discerning an image requested by a user Is Input; the image ID Is transmitted to a second 
communication apparatus; and the image retrieved correspondingly to the image ID Is receh^ed, and by the second 
communication apparatus, the image ID transmitted from the first communication apparatus is received; the image 
comprised of at least one Image of a foreground component image having foreground object components constituting 
a foreground object and a background component Image having background object components constituting a back- 
ground object separated from image data that is obtained by an image pickup element having a predetemnined number 
of pixels, each having a time Integration effect, and that is comprised of pixel values determined for every pixels in 
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accordance with an amount of light forming an image integrated over time, is stored; the image corresponding to the 
received image ID is retrieved from among the stored images; the retrieved image is output to the first communication 
apparatus; and a charge processing Is performed In accordance with the retrieved Image. 

[1044] In the 11^^ communication apparatus and method and the 16^ program of the present invention, an ID dis- 
5 ceming an image requested by a user is input; the Image ID is transmitted to other communication apparatus; and the 
image retrieved con-espondingly to the Image ID is received. 

[1045] In the 12^ communication apparatus and method and the 17^ program of the present invention, an image 
ID transmitted from other communication apparatus is received; the image comprised of at least one image of a fore- 
ground component Image having foreground object components constituting a foreground object and a bacl(ground 

10 component Image having background object components constituting a background object separated from Image data 
that Is obtained by an Image pickup element having a predetemilned number of pixels, each having a time Integration 
effect, and that Is comprised of pixel values determined for every pixels In accordance with an amount of light fonnlng 
an image integrated over time, is stored; the image corresponding to the received image ID is retrieved from among 
the stored images; the retrieved image is output to other communication apparatus; and a charge processing is per- 

IS formed in accordance with the retrieved image. 

[1046] In the 13^ communication apparatus and method and the 18^ program of the present invention, the image 
comprised of at least one Image of a foreground component Image having foreground object components constituting 
a foreground object and a background component image having background object components constituting a back- 
ground object separated from Image data that is obtained by an Image pickup element having a predetennlned number 

20 of pixels, each having a time integration effect, and that is comprised of pixel values detennined for every pixels in 
accordance with an amount of light fomning an image integrated over time, is input; the input image is stored; and a 
payment processing is performed in accordance with the stored image. 

[1047] In the 6*^ communication system and method and the 29*^ program of the present invention, by a first com- 
munication apparatus, the image comprised of at least one Image of a foreground component Image having foreground 

25 object components constituting a foreground object and a background component Image having background object 
components constituting a background object separated from image data that Is obtained by an image pickup element 
having a predetermined number of pixels, each having a time integration effect, and that is comprised of pixel values 
determined for every pixels in accordance with an amount of light fomiing an Image Integrated overtime, is Input; and 
the input image is transmitted to a second communication apparatus, and by the second communication apparatus, 

30 the image transmitted from the first commu nication apparatus is received; the received image are stored; and a payment 
processing is perfomned in accordance with the stored image, in the 14*^ communication apparatus and method and 
the 20*^^ program of the present invention, the Image comprised of at least one Image of a foreground component image 
having foreground object components constituting a foreground object and a background component Image having 
background object components constituting a background object separated from image data that is obtained by an 

35 image pickup element having a predetennined number of pixels, each having a time integration effect, and that is 
comprised of pixel values detenmined for every pixels in accordance with an amount of light forming an image integrated 
overtime, is input; and the input image is transmitted to other communication apparatus. 

[1048] In the IS*' communication apparatus and method and the 21 program of the present invention, an image 
transmitted from other communication apparatus is received; the received Image is stored; and a payment processing 

40 Is performed In accordance with the stored Image. 

[1049] In the 16**^ communication apparatus and method and the 22™^ program of the present invention, a retrieval 
request infomiation of a user is input; the Image comprised of at least one image of a foreground component Image 
having foreground object components constituting a foreground object and a background component Image having 
background object components constituting a background object separated from Image data that is obtained by an 

45 image pickup element having a predetermined number of pixels, each having a time integration effect, and that is 
comprised of pixel values detemiined for every pixels in accordance with an amount of light fomiing an image Integrated 
over time, is input; the image corresponding to the input retrieval request information is retrieved from among the stored 
images; the retrieval result is output; and a charge processing is performed in accordance with the retrieval result. 
[1050] In the 7*^ communication system and method and the 23"* program of the present invention, by a first com- 

so munbatlon apparatus, a retrieval request infomiation of a user is input; the retrieval request infomiation is transmitted 
to a second communication apparatus; and the retrieval result transmitted from the second communication apparatus 
is received, and by the second communication apparatus, the retrieval request infonrnatlon transmitted from the first 
communication apparatus is received; the image comprised of at least one image of a foreground component image 
having foreground object components constituting a foreground object and a background component Image having 

S5 background object components constituting a background object separated from image data that is obtained by an 
image pickup element having a predetermined number of pixels, each having a time Integration effect, and that Is 
comprised of pixel values detemnined for every pixels in accordance with an amount of light f onming an image integrated 
overtime, is stored; the image con^esponding to the received retrieval request information is retrieved from among the 
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stored images; the retrieval result is transmitted to the first communication apparatus; and a charge processing is 
performed in accordance with the retrieval result. 

[1051] In the 1 7*^ communication apparatus and method and the 24^ program of the present invention, retrieval 
request information of a user is input; the retrieval request information is transmitted to other communication apparatus; 
and the retrieval result transmitted from the other communication apparatus Is received. 

[1 052] In the 1 8^ communication apparatus and method and the 25^^ program of the present invention, the retrieval 
request infoimation transmitted from other convnunication apparatus is received; the image comprised of at least one 
image of a foreground component image having foreground object components constituting a foreground object and 
a baclcground component image having background object components constituting a baclcground object separated 
from Image data that is obtained by an Image pickup element having a predetermined number of pixels, each having 
a time integration effect, and that is comprised of pixel values determined for every pixels in accordance with an amount 
of light fomning an image integrated over time, is stored; the Image corresponding to the received retrieval request 
infomiation is retrieved from among the stored images; the retrieval result is transmitted to the other communbation 
apparatus; and a charge processing is performed in accordance with the retrieval result. 

[1 053] In the 1 9^ communication apparatus and method and the 26*^ program of the present invention, foreground 
component image disceming information indicative of a foreground component image having foreground object com- 
ponents constituting a foreground object of image data that Is obtained by an Image pickup element having a prede- 
temnined number of pixels, each having a time integration effect, and that is comprised of pixel values determined for 
every pixels in accordance with an amount of light fomiing an image integrated overtime, is Input; background com- 
ponent image discerning information indicative of a background component image having background object compo- 
nents constituting a background object of the image data Is input; foreground component image positional infonmation 
and background component image positional information corresponding to the foreground component image disceming 
infomiation and the background component image disceming information are generated; and the generated foreground 
component image positional Infomiation and the generated background component image positional information are 
output. 

[1054] In the 8*^ communication system and method and the 27^ program of the present Invention, by a first com- 
munication apparatus, a foreground component image discerning information indicative of a foreground component 
image having foreground object components constituting a foreground object of image data that is obtained by an 
image pickup element having a predetennined number of pixels, each having a time integration effect, and that is 
comprised of pixel values detemnined for every pixels in acconjance with an amount of light fomning an Image integrated 
overtime, is input; a background component image disceming infomiation indicative of a background component image 
having background object components constituting a background object of the image data is input; the foreground 
component Image disceming Infomnation and the background component image discerning information are transmitted 
to a second communication apparatus; and a foreground component image positional Information and a background 
component Image positional information transmitted from the second communication apparatus are received, and by 
a second communication apparatus, the foreground component image discerning information transmitted from the first 
communication apparatus Is received; the background component image disceming information transmitted from the 
first communication apparatus is received; the foreground component image positional infomiation and the background 
component image positional infomiation corresponding to the foreground component image discerning information 
and the background component image discerning Infomiation are generated; and the generated foreground component 
image positional information and the generated background component image positional information are transmitted 
to the first communication apparatus. 

[1 055] In the 20^ communication apparatus and method and the 28*^ program of the present invention, a foreground 
component image disceming Information indk:ative of a foreground component Image having foreground object com- 
ponents constituting a foreground object of image data that is obtained by an image pickup element having a prede- 
tennined number of pixels, each having a time integration effect, and that is comprised of pixel values detemfiined for 
every pixels in accordance with an amount of light fomnlng an image integrated over time, is input; a background 
component image discerning infomrietion Indicative of a background component image having background object com- 
ponents constituting a background object of the image data is input; the foreground component image discerning 
infonnation and the background component image discerning information are transmitted to other communication ap- 
paratus through a networi<; a foreground component image positional information and a background component image 
positional infomiation transmitted from the other communication apparatus through a network are received. 
[1056] in the 21 communication apparatus and method and the 29^ program of the present invention, the foreground 
component Image discerning information of image data transmitted from other communication apparatus through a 
network is received, the image data being obtained by an image pickup element having a predetermined number of 
pixels, each having a time integration effect, and that is comprised of pixel values determined for every pixels in ac- 
cordance with an amount of light fomiing an image Integrated overtime; the background component image disceming 
infonnation transmitted from the other communication apparatus is received; a foreground component image positional 
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information and a bacl<ground component image positional infonmation corresponding to the foreground component 
image discerning information and the background component image discerning information are generated; and the 
generated foreground component Image positional information and the generated background ccnnponent image po- 
sitional information are transmitted to the other communication apparatus via the network. 
5 [1057] As a result, it is possible to encrypt an image by adding motion blur to the image, and also to decrypt the 
encrypted image by eliminating the motion blur, that is, adding motion biur Inverse to the motion blur added in the 
encryption process. 

[1058] Further, it is possible to obtain a clear foreground component image having been eliminated with motion blur 
by adjusting the motion biur contained in the foreground component image, and to generate a synthesized image 
10 appearing seemingly natural by combining the motion blur adjusted foreground component Image with a background 
component image. As a result, by adjusting the motion blur in the foreground component image and by synthesizing 
an Image by processing the resulting foreground component with a background component, tt is possible to correct 
images more naturally 

[1 059] Moreover, since the image obtained by separating, synthesizing or connecting an image on the basis of mixed 
15 state of the image, can be purchased or sold via the network 1 , the separated, synthesized or corrected image can be 
effectively utilized by selling the images processed by the users. 

[1060] Furthennore, it is possible to retrieve tlie Images which are being photographed, or have been photographed 
and stored by the camera terminal units 28-1 to 28-n by a sinnple Input of the retrieval conditions. 
[1 061] Moreover, by storing Images in a predetemnined storage server rather than In user's computer, and by reading 
^ out and utilizing the images on the basis of positional infomnatioh of the storage server as needed, pre-existing image 
data can be called out and utilized as needed without maintaining them in their own communication apparatus, such 
that images can be efficiently utilized. 

[1062] According to above descriptions, it is possible to encrypt image by adding motion blur to the Image and decrypt 
the image by eliminating the motion blur. Further, it is also possible to eliminate motion blur to generate a clear image 

25 and correct an image more naturally to synthesize an image In which motion blur is adjusted by combining the motion 
blur adjusted foreground component with a background image. Further, it is possible to effectively utilize via the network 
images separated, synthesized or corrected on the basis of mixed state of the image. Moreover, it is also possible to 
retrieve the separated, synthesized or corrected images via the network. In addition. It is possible to call out and utilize 
pre-existing images without maintaining them In their own communication apparatus. 

30 [1063] The recording medium having a program for performing signal processing according to the present invention 
may be, as shown in FIG. 2 and FIG. 3, a package medium having magnetic discs 61 , 91 (including flexible discs), 
optical discs 62, 92 (including CD-ROM (Compact Disc-Read Only Memory) and DVD (Digital Versatile Disc), magneto- 
optical discs 63, 93 (Including MD ((Mini-Disc)(trademark)), semiconductor memories 64, 94, etc. recorded with the 
program for being able to be distributed to the users as an individual component separated from a computer. To be 

35 provided to the users by having already Incorporated in a computer, the above recording medium may also be a ROM 
42, 72 in which the program is recorded, or a hard disc included in the storage units 48, 78. 
[1064] Further, in the present specification, the steps for describing the program recorded in the recording medium 
comprise the processing to be performed in time sequence in the order of description and the processing to be per- 
fomned simultaneously or separately, not necessarily in time sequence. 

40 [1065] Furthenmore, In the present specification, a system means the whole apparatus comprising a plurality of ap- 
paratuses. 

Industrial Applicability 

45 [1 066] According to the present invention, it is possible to improve the image processing speed and to reduce cost 
for service by distributing the image processing to perform only the requested processing. 



Claims 

50 

1 . A communication system comprising: 

separating means for separating predetemriined image data that is obtained by an Image pickup element having 
a predetennined number of pixels, each having a time integration effect, and that is comprised of pixel values 
55 detennined every pixel In accordance with an amount of light fomning an image integrated over time into a 

foreground component image having foreground object components constituting a foreground object and a 
background component image having background object components constituting a background object; 
requested Infomriation input means for inputting requested information of a user; and 
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encoded data output means for outputting encoded data generated when said image is separated into said 
foreground component image and said background component image by said separating means, on the basis 
of said requested infonnation input by said requested infomiation input means. 

2. The communication system as claimed In claim 1 . further comprising charging means for perfomiing charge 
processing in accordance with said requested Infomnatlon. 

3. The communication system as claimed in claim 2, wherein. In accordance with said requested Infomiation, said 
charging means generates charge infomiatton including an ID of said user, an ID of said communication system, 
and cost information con-esponding to said requested infonmation. 

4. The communication system as claimed in claim 3, wherein said charging means performs the charge processing 
with respect to a financial account of said user on the basis of said charge Information. 

5. The communication system as claimed in claim 4. wherein said charging means performs the charge processing 
by deducting the number of points corresponding to said cost information and user used In the charge processing 
from the number of points of each. 

6. The communication system as claimed in claim 2, wherein after said charging means has finished the charge 
processing, said encoded data output means outputs said encoded data to a communication apparatus of said 
user after finishing the charge processing. 

7. The communication system as claimed in claim 6, wherein said requested Information Input means inputs said 
predetemnlned image data in addition to said requested infomnation of said user; and 

said encoded data output means outputs said encoded data as significant data generated when said prede- 
temriined image is separated Into said foreground component image and said baclcground component image by 
said separating means on the basis of said requested information input by said requested information input means 
and said predetermined image data, in a way obtainable only to said user after the charge processing by said 
charging means has finished. 

8. The communication system as claimed In claim 7, further comprising: 

an area Infomnatlon generating means for generating area information for discerning any of a foreground area 
having foreground object components constituting a foreground object of said predetemriined Image data, a 
background area having background object components constituting a background object of said predeter- 
mined image data and a mixed area in which said foreground area and said background area are mixed, 

wherein said significant information includes said area information, and 

wherein said encoded data output means outputs said encoded data as said area information generated 
when said predetermined image is separated Into said foreground component image and said background com- 
ponent image by said separating means on the basis of said requested infomnatlon input by said requested infor- 
mation input means and said predetermined image data, in a way obtainable only to said user after the charge 
processing by said charging means has finished. 

9. The communication system as claimed in claim 7, further comprising: 

a mixture ratio generating means for generating a mixture ratio indicating the ratio at which said foreground 
area and said background area are mixed in said mixed area of said predetennined image data, 

wherein said significant infomnation includes said mixture ratio, and 

wherein said encoded data output means outputs said encoded data as said mixture ratio generated when 
said predetermined image is separated into said foreground component Image and said background component 
Image by said separating means on the basis of said requested infonnatlon input by said requested infomiation 
input means and the predetennined image data, in a way obtainable only to said user after the charge processing 
by said charging means has finished. 

10. The communication system as claimed in claim 7, wherein said separating means separates said predetennined 
Image data that is obtained by the image pickup element having the predetermined number of pixels, each having 
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a time integration effect, and that is comprised of the pixel values detennined eveiy pixel in accordance with the 
amount of tight forming a predetermined Image integrated overtime into said foreground component image having 
the foreground object components constituting the foreground object and said baclcground component image hav- 
ing the background object components constituting the baclcground object on the basis of said area infomfiation 
and said mixture ratio, and 

wherein said significant infomiation includes said foreground component image and said background com- 
ponent image, and 

wherein said encoded data output means outputs said requested infomiation input by said requested infor- 
mation input means and said encoded data as said foreground component innage and said background component 
image generated when said predetermined image is separated into said foreground component image and said 
background component image by said separating means, in a way obtainable only to said user after the charge 
processing by said charging means has finished. 

1 1 . The communication system as claimed in claim 6, wherein said requested infomnation Input means inputs an image 
data ID for disceming said predetemnined image data in addition to said requested information of said user, and 

wherein said encoded data output means outputs said encoded data as said signif teant inf omnation generated 
when said predetermined image is separated into said foreground component image and said background com- 
ponent image by said separating means on the basis of said requested information input by said requested infor- 
mation input means and the predetemntned image data corresponding to said image data ID, in a way obtainable 
only to said user after the charge processing by said charging means has finished. 

12. The communication system as claimed in claim 11 , further comprising: 

area infonmation generating means for generating area infomnation for disceming any of a foreground area 
having foreground object components constituting the foreground object of the predetemnined image data 
corresponding to said image data ID, a background area having background object components constituting 
the background object of the predetermined image data corresponding to said image data ID, and a mixed 
area In which said foreground area and said background area are mixed, 

wherein said significant infomnation includes said area information, and 

wherein said encoded data output means outputs said encoded data as said area information generated 
when said predetemiined image Is separated Into said foreground component image and said background com- 
ponent image by said separating means on the basis of said requested infomnation input by said requested infor- 
mation input means and said predetennined image data, In a way obtainable only to said user after the charge 
processing by said charging means has finished. 

13. The communication system as claimed In claim 11 , further comprising: 

mixture ratio generating means for generating a mixture ratio of said mixed area in the predetermined Image 
data corresponding to said Image data ID, 

wherein said significant information includes said mixture ratio, and 

wherein said encoded data output means outputs said encoded data as said mixture ratio generated when 
said predetenmined image is separated into said foreground component image and said background component 
image by said separating means on the basis of said requested information input by said requested information 
input means and the predetermined image data con-esponding to said image data ID, in a way obtainable only to 
said user after the charge processing by said charging means has finished. 

14. The communication system as claimed in claim 11 , wherein said separating means separates the predetemiined 
image data that is obtained by the image pickup element having the predetemiined number of pixels, each having 
a time integration effect, and that is comprised of pixel values determined every pixel In accordance with the amount 
of light fonming a predetemiined image Integrated over time into said foreground component image having the 
foreground object components constituting the foreground object and said background component Image having 
the background object components constituting the background object on the basis of said area information and 
said mixture ratio, 

wherein said signifteant information Includes said foreground component image and said background com- 
ponent image, and 

wherein said encoded data output means outputs said requested Infomiation input by said requested Infor- 
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mation input means and said encoded data as said foreground component image and said baclcground component 
image generated when said predetemilned image is separated into said foreground component image and said 
background component image by said separating means, in a way obtainable only to said user after the charge 

processing by said charging means has finished. 

15. A communication method comprising: 

a separating step of separating image data that is obtained by an image pickup element having a predetermined 

number of pixels, each having a time integration effect, and that is comprised of pixel values detemnined every 

pixel in accordance with an amount of light forming an image integrated over time Into a foreground component 

image having foreground object components constituting a foreground object and a background component 

image having background object components constituting a background object; 

a requested information inputting step of inputting requested information of a user; and 

an encoded data outputting step of outputting encoded data generated when said image is separated into said 

foreground component image and said background component image in said separating step on the basis of 

said requested infonnation input in said requested information inputting step. 

16. A recording medium having a computer- readable program, wherein the program comprises: 

a separation controiling step of controliing the separation of image data that is obtained by an image pickup 
element having a predetenmlned number of pixels, each having a time Integration effect, and that is comprised 
of pixel values detemnined every pixel in accordance with an amount of light fomning an image integrated over 
time into a foreground component Image having foreground object components constituting a foreground ob- 
ject and a background component Image having background object components constituting a background 
object; 

a requested Infomnatlon Input controlling step of controlling the input of requested (nfonnation of a user; and 
an encoded data output controlling step of controlling the output of encoded data generated when said image 
Is separated into said foreground component image and said background component image In said separation 
controlling step on the basis of said requested infomnation input in said requested infomnation input controlling 
step. 

17. A program for a computer comprises: 

a separation controlling step of controlling the separation of image data that is obtained by an Image pickup 
element having a predetermined number of pixels, each having a time integration effect, and that is comprised 
of pixel values detennined every pixel in accordance with an amount of light fomiing an Image integrated over 
time into a foreground component image having foreground object components constituting a foreground ob- 
ject and a background component image having background object components constituting a background 
object; 

a requested Information Input controlling step of controlling the input of requested infonmation of a user; and 
an encoded data output controlling step of controlling the output of encoded data generated when said Image 
Is separated into said foreground component Image and said background component image in said separation 
controlling step on the basis of said requested information input in said requested information input controlling 
step. 
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